44 Commits

Author SHA1 Message Date
vh c37a425276 feat(booth): prev/next arrows in the image viewer
Zooming an image now shows ‹ / › arrows at the left/right edges that step
to the previous/next image in the booth (gallery sorted-rel order), wrapping
around, plus keyboard ←/→. Arrows are hidden when a booth has a single image.
booth_view_file computes neighbors via a new booth_image_names() helper and
passes prev_url/next_url to view.html. 3 new tests, suite 47 passing;
deployed + verified live on nh3-dev :8090.
2026-08-05 01:30:30 -07:00
vh 315faac4b5 feat(booth): view .md (rendered) and .txt/.log in-booth without downloading
Loose .md/.txt/.log files rendered as forced-download links in the gallery
and downloaded (or showed raw) when opened. Now they open in a readable
in-booth page via the existing /b/<name>/view route:
  - .md  -> rendered HTML (Python-Markdown: fenced code, tables, sane lists),
           styled in an Australis .markdown-body with the viewer chrome;
  - .txt/.log -> preformatted <pre> text view.
The gallery links docs to the viewer (📄) instead of a download; the view
page keeps a ⬇ (?dl=1) for saving. Files over 2 MB hand back raw. New
markdown dep (optional-import: degrades .md to text view if absent).
booth_image_view -> booth_view_file (now handles image + doc + raw-fallback).
9 new tests, suite 44 passing; deployed + verified live on nh3-dev :8090.
2026-08-05 01:13:38 -07:00
vh b5ae9365ff memory: triton backend is a HARD NO on the Ampere comfy box (fp8e4nv), deferred to Ada
Recorded under Tried-and-abandoned: --enable-triton-backend on irv-ml1's
A6000 crashes every render (triton's fp8 dequant needs fp8e4nv/e4m3, which
sm_86 lacks; the global --fp8_e4m3fn-text-enc dequant dies at CLIPTextEncode,
taking fp8 and int8 alike). Reverted cleanly; cu130 rebuild won't fix a
hardware format limit; revisit at the Ada refresh. Also captures the
compose-env-needs-recreate-not-restart mechanics lesson.
2026-08-03 21:29:00 -07:00
vh 348c5c12a2 chore(booth): gitignore uv.lock (stray artifact from uv run; booth tracks no lockfile) 2026-08-03 13:06:02 -07:00
vh 8577e7e248 feat(booth): wrap verbatim index.html booths with a back-to-booths chip + inherited favicon
Verbatim-index.html booths were served raw (FileResponse) with no base
template, so they had no favicon and no way back to the booth index — the
gap the app-rendered gallery/zoom pages already covered via base.html.

booth_view now reads a small verbatim index.html and, via a pure
wrap_verbatim_html(), injects:
  - a fixed-position 'all booths' chip (scoped class, max z-index, hidden
    in print), pinned top-right (empty on left-aligned report layouts; a
    top-left chip clips the page title) and appended at the END of the
    document so it never reorders the page;
  - the Booth favicon at the first head-ish seam, only if the page declares
    no icon of its own.

Injection is doctype/charset-safe for the compact HTML real booths use
(<!doctype html><meta charset><title><style>…content, no explicit head/
body): nothing is ever placed ahead of a leading <!doctype> (which would
force quirks mode), and the ~250B favicon link keeps the charset <meta>
inside the first-1024-byte detection window. The raw file route
(/b/<name>/index.html) stays byte-for-byte, so assets and ?dl=1 downloads
are unchanged; files over 8 MB serve raw, unwrapped.

Verified live on nh3-dev :8090 across the real booth shapes (compact-HTML
crow-*/jackdaw-*/mimir-favicon, well-formed dcc-summarizer-ab, own-icon
edict-favicon). 10 new tests; suite 38 passing.
2026-08-03 13:05:30 -07:00
vh d4d9956fed memory: correct extdev sudo status in 07-25 herald-install entry
The [2026-07-25] nh3-extdev herald-install entry claimed infra-ops was
sudo-less on extdev, explaining the use of the lkraven@ NOPASSWD path.
That parenthetical was wrong: infra-ops has had full NOPASSWD sudo on
extdev since 2026-06-25. Kept the historical action (lkraven@ was used)
and appended a dated correction so a future session self-serves as
infra-ops instead of taking the lkraven@ hop unnecessarily. Surfaced
during the 2026-08-03 herald.py:363 re-nudge-bug arc.
2026-08-03 08:01:17 -07:00
vh bc084edbee memory: snapshot for /clear — worldtree b168/#384/#385 arc complete
Collapsed the completed-arc bloat out of Current state / in-flight (line 125:
6487->1073 chars; runbooks moved to persistent-memory.d/2026-08-03-worldtree-b168-384-385-arc.md).
Added the 2026-08-03 arc Recent-decisions entry. Auto-archived 20 oldest
Recent-decisions entries (2026-07-08..07-14) to archival-memory.md (index 345->308
lines). Refreshed the b169-deploy watch, unpushed-count, and handoff.
2026-08-03 00:12:59 -07:00
vh 4be87f1a94 memory: b169 pull-fail was a shared-containerd race, NOT disk; dont prune in-use img
corviduo-dev demo+personal+pinned share one containerd; concurrent pull of b169's
torch layer failed personal mid-race while demo completed. Image 6e34a87 fully
valid (demo running it healthy). Fix = re-run. NEAR-MISS: prune/rmi 6e34a87 would
have downed demo. Lesson: docker ps running images before any prune cleanup.
2026-08-02 23:57:48 -07:00
vh 45594e3891 memory: config-delta pre-sync rule — only config/ is bind-mounted, agents/ in-image
Verified via docker inspect: worldtree containers bind-mount only config/ host-side
(providers/model_roles/matrix/policies/defaults/env.public = pre-syncable set);
agents/ (schemas.yaml) + code ship in-image. So b169's schemas.yaml delta needs
no host pre-sync; only config/*.yaml changes are boot-blocking-pre-syncable.
2026-08-02 23:40:39 -07:00
vh 523b28f12f memory: production sweep DONE — 785 April DCC dupes deleted, arc complete
Operator-approved: backed up 785 rows (reversible) -> deleted job_id=b59c147c5ce0
from main (4009->3224, target cleared) -> bounced api (HNSW reload) -> confirmed
search fiction-only. #384/#385 closed, worldtree arc complete.
2026-08-02 23:12:21 -07:00
vh b4ef0600d2 memory: BrokenPipe wing-git watch CLOSED (round-2 commits clean, chown enabled)
worldtree-dev closed the watch with git evidence: both round-2 wing commits
landed clean under exec -u 1000; loose-object chown enabled it. Self-healed as
predicted; last thread of the reindex-as-root fallout closed.
2026-08-02 22:54:32 -07:00
vh 3a829890c5 memory: worldtree #384/round-2 arc COMPLETE + #381 restart fired
Both books terminal (DCC 705 / P&P 667 indexed, 0 failures, #385 budget fix
validated); #381 restart-after-ingest fired, api healthy, wing retrievable
(search_library returns DCC+P&P from fiction). ratatoskr verify handed off.
Delete-sweep precondition now met (stale main rows = genuine duplicates).
2026-08-02 22:53:01 -07:00
vh d72597bade memory: /health W_OK blind-spot for root-owned job subdirs (muninn-dev)
os.access(ingestion_root, W_OK) tests only the root dir, so a root-owned job
subdir passes requeue guards + /health both while being unclaimable by uid 1000.
muninn-dev added an ownership column to the post-move check.
2026-08-02 22:47:51 -07:00
vh 8bc5be35ce memory: reindex-as-root contaminated KB tree; adopt docker exec -u 1000 rule
sudo docker exec --reindex ran as root (app=uid 1000), leaving a root-owned
.old- dir (blocked uid-1000 rmtree) + 60 root-owned git objects. Fixed: rm -rf
.old- (tar'd first) + chown 1000:1000 the objects. RULE: docker exec that writes
pipeline state uses -u 1000. GOTCHA: don't head a scope-defining find (.old- had
153 files, head-20 hid the 60 git objects).
2026-08-02 22:45:33 -07:00
vh 3790669fb5 memory: round-2 sequencing (mimir-dev drives, muninn-dev confirms, infra-ops #381)
muninn-dev HOLD LIFTED (guard-verified 02:36Z). Recorded who owns what:
round-2 requeue = mimir-dev browser flow (op ruling pending); muninn-dev = gate
confirmer / post-move board-check custody; infra-ops = #381 restart after both
terminal + later supervised sweep.
2026-08-02 19:40:04 -07:00
vh 9bd5f2a4b1 memory: mv -t is the real guard (trailing slash is a false guard); dont-tidy note
muninn-dev tested on coreutils 9.1: mv src failed/ with failed/ missing STILL
silently renames (rc=0) — trailing slash buys nothing. mv -t <dir> <src> refuses
a missing target loudly (rc=1). Adopted as house convention. Also: DCC+P&P rest
in failed/ with manifest=complete until round-2 (requeue keys on dir placement) —
deliberate, don't tidy.
2026-08-02 19:38:08 -07:00
vh 4eb2724712 memory: add silent-failure-mode note to the mv lesson (muninn-dev)
A misplaced ingestion-state move is silent (list_jobs OK, files inert; sole
symptom = job absent from board, job_row->None/404). Verify job is on the board
+ job.dispatch.json survived after any move; don't trust mv exit codes.
2026-08-02 19:36:06 -07:00
vh f8d0f3081d memory: lesson — mv into nonexistent failed/ renamed the job dir
worldtree-dev round-2 unblock: mv job complete/->failed/ assumed failed/
existed; it didn't on personal, so mv renamed job1 to 'failed' + nested job2.
Caught + reconstructed via complete/ scratch, no data loss. Lesson: verify dir
exists with [ -d ] before mv-into (empty ls is ambiguous); guard peer fs cmds
with mkdir -p.
2026-08-02 19:30:18 -07:00
vh 11f9856cd5 memory: wing-git = self-healing watch; pending prod delete-sweep heads-up
worldtree-dev accepted #384 recovery. Wing-git BrokenPipeError ruled a
watch-item (self-heals on DCC round-2 re-run; escalate only if 2-in-a-row).
Recorded a heads-up for a future production delete-by-job_id of 785 April-era
DCC orphans (job_id=b59c147c5ce0, 'main' collection) — to be handled supervised
(backup + scope-confirm + operator-in-loop) after ratatoskr's verify.
2026-08-02 19:17:56 -07:00
vh c3b6630684 memory: #384 recovery on personal DONE (reindex+restart+verify)
b168 preflight-confirmed live; reindex exit0 (97 concepts); #381 restart
healthy; metadata verify 5/5 fiction rows carry note_id+path. Open item:
wing git commit BrokenPipeError (kb_ingest_degraded, reconcile-visible) —
worldtree-dev's call. Recorded chroma-verify runbook (docker exec -i).
2026-08-02 19:15:39 -07:00
vh 62d5a45182 memory: b168 pushed + pending #384 reindex/#381 restart on personal
worldtree-dev pushed the train (main + tags v1.0.0b168/staging) and verified my
pre-sync matches the commit byte-for-byte. Recorded the next pending op: after
b168 CI-lands on personal, run --reindex of job mimir-6351554e8e8f + #381 restart
on their exact commands.
2026-08-02 19:03:12 -07:00
vh e0b616b946 memory: providers.yaml summarization pre-sync DONE (#384/#385 b168)
Synced the boot-gating summarization block + deep-reasoning desc to demo +
personal via deploy-wt-config (health-gated); replied GO. Recorded gotchas:
unpushed-commit fetch falls back to default branch (verify SHA exists), and
paste-sourced YAML needs indent/validate (8-space desc was invalid).
2026-08-02 19:01:30 -07:00
vh b901468752 memory: track pending providers.yaml summarization pre-sync (#384/#385)
worldtree-dev advance notice: #384/#385 train adds a boot-blocking
'summarization' capability block (vastblueai-gateway/gen) that in-image
muninn/config.yaml references. Must pre-sync into providers.yaml on demo
+ personal via deploy-wt-config BEFORE the image deploy, from the exact
commit. ACKed; recorded as an open watch.
2026-08-02 18:11:28 -07:00
vh a713d3f2c4 memory: #381 restart DONE — fiction wing retrieval-visible on personal
DCC job mimir-6351554e8e8f complete; restarted worldtree-personal-worldtree-api-1
(healthz/readyz 200); Mimir search_library on wings=[fiction] returns 10 DCC hits
(retrieval visibility confirmed, not grounding — ratatoskr runs grounding verify).
Recorded the Mimir-session probe runbook (SSE stream, tool-events).
2026-08-02 14:41:31 -07:00
vh f30c41409f memory: track pending #381 restart (DCC re-ingest, worldtree-dev)
worldtree-dev gave advance notice: standing #381 restart of
worldtree-personal-worldtree-api-1 will be requested when DCC re-ingest
job mimir-6351554e8e8f (fiction wing) goes terminal in a few hours.
ACKed no scheduling constraints; recorded as an open watch so a fresh
session catches the request.
2026-08-02 13:24:31 -07:00
vh 9b2c47602d memory: donut onyx-58 multi-clip ref reverted; emotion-away-from-clone lesson
A/B (5 pinned-seed neutral pairs) showed original single-clip seg000
(16.3s) beats the 52s 4-take concat on timbre fidelity — reverted live +
build-source to seg000-alone. Recorded two craft lessons: more reference
audio is not automatically better when takes vary; emotion steering pulls
output away from a cloned voice fast (keep clones emotion-neutral).
2026-08-02 13:22:41 -07:00
vh cac381a114 memory: donut voice expanded with onyx-58 clips (16.3s->52s ref)
Folded the onyx-58 Booth bundle (seg101/seg110/seg148, all Princess Donut
per operator-confirmed misdiarization of seg148) into the Zonos gateway
donut voice reference alongside the original 65-frost seg000. New ref =
52.0s @ 44.1kHz mono s16, deployed to irv-ml1 live bind-mount +
build-source tree; old ref backed up. A/B audition in booth donut-onyx58.
2026-08-02 13:14:12 -07:00
vh 1ab5465293 memory: snapshot — mimir-inbox/#377 read-path arc (deploy + #380/#381/#382 + donut voice); archived 11 old entries 2026-08-02 07:33:49 -07:00
vh a02ef5d851 docs: note worldtree #383 tier3_wings parity-fold pending (next config sync) 2026-08-01 19:39:26 -07:00
vh dfcf223ff1 docs: park Audio8 TTS-zoo addition (operator, non-blocking) 2026-08-01 17:34:51 -07:00
vh 8a824f85a3 docs(muninn-gate): Dockerfile comment — dispatch pin now 0.1.5, SHA-tag note
Gate rebuilt off vh/muninn-gate main (bc04c4c) for the muninn-dispatch
0.1.4->0.1.5 bump (concept_schema/concept_schema_source row fields). Gate
version unchanged at 0.0.14; note to also tag the image with the source SHA
for traceability, and that the pin is authoritative in pyproject.
2026-08-01 17:27:14 -07:00
vh 0441995ac8 docs(muninn-gate): sync example to reality — mimir-inbox key control scope + staging live (#377)
The mimir-inbox deploy changed two things the committed example documents:
the mimir-inbox key gained the control scope (2026-08-01, for cancel/retry),
and the staging root is no longer a placeholder — it's the real shared dir on
corviduo-dev, path-agreement probe PASS (muninn-dev). Bind was already correct
at :8090 (the stale :8080 was only in the gate repo's own example).
2026-08-01 14:36:31 -07:00
vh 4b54a32d64 feat(mimir-inbox): book-ingestion UI stack on corviduo-dev:8091 (#377)
WG-internal FastAPI+HTMX front end for large-document ingestion into the
Muninn KB, over the muninn-gate API (browser -> mimir-inbox -> staging ->
path-addressed POST /jobs). Co-located on corviduo-dev with the gate (:8090)
and the worldtree-personal muninn watcher per the operator's 2026-08-01
co-location ruling (reversing the earlier off-box/NFS plan; worldtree-dev
approved the box placement).

- Dockerfile: python:3.11-slim + uv sync --no-dev --frozen (--no-dev is
  load-bearing; the dev group's muninn-dispatch path source is absent in-image
  and INV-MI-7 forbids importing it). Single-stage by design — src/ stays in
  the final image (uv installs the project editable-linked to src/).
- compose.yaml: uid 1000, host-net bind 10.250.50.152:8091, staging :rw,
  TCP-liveness healthcheck (deliberately not coupled to gate reachability).
- Built from vh/mimir-inbox HEAD c8ab38f; deployed + healthy.

Records the open-in-place claim semantics (worldtree-dev, runner.py:362-367)
and the INV-MI-19 retention rule (staged files persist until job terminal;
gate retry returns a false-200 on a swept source) in persistent memory.
2026-08-01 14:31:14 -07:00
vh e6eabc1dde memory: snapshot — session close (muninn-gate #377 deployed on corviduo-dev, wtsdk 1.1.0 published + vh token, kimi-k3 reasoning-cap fix relayed to heid, magpie TTS rejected) 2026-07-31 21:27:13 -07:00
vh 786462ac9c feat(muninn-gate): WG-internal Muninn ingestion front door stack (#377)
Deployed on corviduo-dev, co-located with the worldtree-personal muninn
watcher. ingestion_root=/data/state/ingestion (shared state volume, byte-
identical to the watcher); runs as uid 1000 to write the queue; staging
bound :ro at the ratified /mnt/muninn-staging/mimir-inbox (local placeholder
until the shared mount + mimir-inbox writer land). Boot verified: /ping
{"service":"ok"}, /health watcher.running=true (byte-identity proven).

Image built out-of-band with the Gitea read token as a BuildKit secret.
Real config (bearer keys) lives on-server at /opt/docker/conf 0600.
2026-07-30 21:33:29 -07:00
vh 17d776fc90 memory: snapshot — multi-day infra session close (config repo + boundary, Zed FIM route, muninn #377, Kimi, herald, Booth v0.1.4) 2026-07-30 06:39:57 -07:00
vh 792aa2852c docs(zed-fim-proxy): source-IP allowlist stays OFF by design (Zed roams WG 10.0.0.0/8) 2026-07-27 22:58:22 -07:00
vh a300cdcd26 feat: Zed edit-predictions keyless FIM route (Qwen2.5-Coder-1.5B / coder-fast)
Deep-research-picked Qwen2.5-Coder-1.5B (BASE, Apache-2.0, native FIM) as a
low-latency inline-completion seat:
- stacks/vllm: vllm-coder service (ana-ml2 GPU1 :8020) + granite shrunk
  (util 0.27->0.13, max-len 131072->16384, seqs 1024->256; granite phasing out)
  to free GPU1 room.
- stacks/litellm: coder-fast alias -> :8020 (mode: completion, /v1/completions).
- stacks/zed-fim-proxy (NEW): keyless /v1/completions front door on ana-docker
  :4141 for Zed (which can't send an auth header) — POST + path + model
  allowlist, injects a coder-fast-scoped virtual key -> LiteLLM :4000. Anon
  /ping liveness. Verified keyless FIM end-to-end.

Zed api_url = http://10.250.50.70:4141/v1, model coder-fast, prompt_format qwen.
Source-IP allowlist off pending the Mac's observed source IP.
2026-07-27 22:55:21 -07:00
vh 8822a0bb81 memory: Worldtree #377 CLOSED — muninn watcher acceptance passed 2026-07-27 08:35:08 -07:00
vh 038e455897 memory: #377 durability resolved (COMPOSE_PROFILES=muninn) + env.public config-as-code extension 2026-07-27 08:27:47 -07:00
vh e06a96fc3e memory: Muninn ingestion-watcher sidecar deployed on personal Worldtree (#377) 2026-07-27 08:21:05 -07:00
vh 8944531ba0 memory: jackdaw-compose.service decommissioned (AI Composer cut from JackDAW v1) 2026-07-27 06:59:21 -07:00
vh 557d0b56d9 memory: demo BIFROST_CLIENT_ALLOWED_HOSTS delta (wyrd-dev provider) + env-change recreate/image-pin foot-gun 2026-07-26 22:17:20 -07:00
vh a5e2d91dfd memory: record nh3-extdev herald install + Booth v0.1.4 download feature 2026-07-25 17:49:36 -07:00
59 changed files with 1668 additions and 343 deletions
+410 -1
View File
@@ -1316,4 +1316,413 @@ _76 older entries archived to archival-memory.md._
- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — but `ssh
infra-ops@ana-docker` HAS NOPASSWD root.** A `sudo cp` as lkraven silently failed → nearly punted
the rest-server recovery. Reach for `infra-ops@ana-docker` for sudo ops.
_Archived 2026-07-08._
_Archived 2026-07-08._
## Archived 2026-08-02 — Recent decisions (archived)
### 2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session
- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` +
`envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal),
YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my
parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.)
_Archived 2026-08-02._
### 2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b
- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker
docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB
registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the
2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to
remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because
daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never
reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd
opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`,
`rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the
frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime
(zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares
`live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start,
never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers
once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon;
log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running
containers).
_Archived 2026-08-02._
### 2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped
- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into
gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17,
vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack
(01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829);
max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and
`repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture
no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone
voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514).
**Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not
A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s).
_Archived 2026-08-02._
### 2026-07-09-granite-gen-memory-extractor-bind-host-synced-on
- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed,
#335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides
intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on
`summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit
both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen**
(b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted
`worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL
NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1`
after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config
INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven
safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed).
StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON:
the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never
updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) —
pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass`
parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider
description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix
(operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING
defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs
when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway
config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD;
live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09):
character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving
character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping
principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the
dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent
but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44,
UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly
from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) +
mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's
`load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character +
character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window
VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev
to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked
config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly
re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard
+ character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning
downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying
(both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37**
(char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09):
mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx
131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45
canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push
of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity
block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev),
monitor armed. See [[reference_corviduo_dev_emergency_ops]].
_Archived 2026-08-02._
### 2026-07-09-granite-gen-memory-extractor-bind-green-lit-for
- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).**
Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty
idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching).
Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08).
**This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the
operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread
01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]].
_Archived 2026-08-02._
## Archived 2026-08-02 — Tried and abandoned (archived)
### 2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s
- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized"
is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but
fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push
finishes. FIX: re-run once built, or gate on `workflow_run: completed`.
_Archived 2026-08-02._
### 2026-07-04-on-prem-t1-train-that-keeps-any-ana
- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe
ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast
on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the
566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`.
_Archived 2026-08-02._
### 2026-07-04-a-systemd-user-daemon-that-shells-out-to
- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools
needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The
althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO
routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`.
_Archived 2026-08-02._
### 2026-07-09-fp8-breaks-morpheus-audio-token-generation
- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC
frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks-
TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the
bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real.
_Archived 2026-08-02._
### 2026-07-09-sentence-chunking-tts-loses-prosody
- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that
spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning
(penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path).
_Archived 2026-08-02._
### 2026-07-09-hf-whisper-datasets-aren-t-actually-whispered
- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8
(not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the
cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) —
that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting.
_Archived 2026-08-02._
## Recent decisions (archived 2026-08-03 batch)
### 2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed
- `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug.
_Archived 2026-08-03._
### 2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt
- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section.
_Archived 2026-08-03._
### 2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning
- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`.
_Archived 2026-08-03._
### 2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code
- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]].
_Archived 2026-08-03._
### 2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause
- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528.
_Archived 2026-08-03._
### 2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway
- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1.
_Archived 2026-08-03._
### 2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s
- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot.
_Archived 2026-08-03._
### 2026-07-13-wt-355-residual-300s-hang-localized-to-our
- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]**
_Archived 2026-08-03._
### 2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree
- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize.
_Archived 2026-08-03._
### 2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min
- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate).
_Archived 2026-08-03._
### 2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on
- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated).
_Archived 2026-08-03._
### 2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree
- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev
R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml`
(`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347
`session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's
personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based
`affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347.
R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe
→ present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string
insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own
yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows
baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo
untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they
want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]].
_Archived 2026-08-03._
### 2026-07-10-comfyui-v0-27-1-success-on-irv-ml1
- `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1,
SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch
2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt`
FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came
only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt`
as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any
transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment).
`docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB
migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3
needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a
mmartial ComfyUI version bump, `git diff <old> <new> -- requirements.txt` — if torch is unpinned, bump the CODE
without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid
1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See
[[reference_irv_ml1_comfyui_mmartial]].
_Archived 2026-08-03._
### 2026-07-10-comfyui-0-25-x-bump-on-irv-ml1
- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev
requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`,
0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the
VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` =
`/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump
broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch
2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke
the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot
`/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention),
re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at
`/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):**
comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention
rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY
the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) →
rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1),
lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.
_Archived 2026-08-03._
### 2026-07-10-biweekly-open-weight-releases-scan-cron-set-up
- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable
systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00`
America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh`
runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in
`~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH
footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS
new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode
ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0):
web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input +
caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle
brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor);
registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired
step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default
+ `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled
1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending.
**NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start
brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session
handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 +
ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token
feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo).
_Archived 2026-08-03._
### 2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b
- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD
Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.**
Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0
KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf);
(4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a
DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over
RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp
template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the
live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively
opens `<think>`/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap
b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).**
_Archived 2026-08-03._
### 2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated
- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.**
Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision,
util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps
ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled
refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path);
Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting
("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway
sampling-defaults wiring still PENDING.
_Archived 2026-08-03._
### 2026-07-08-dpo-was-silently-running-3-epochs-harness-gap
- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3
epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass
`num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs
+ _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the
off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a
~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run.
FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig,
float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`.
GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min
train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step;
train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45
banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0):
ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is
COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral
literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift,
arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep
on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval +
the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO
adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).
_Archived 2026-08-03._
### 2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes
- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.**
Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path
finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied
VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before
`from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does
`model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers
5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the
DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED
un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO
confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93%
util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit.
gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.
_Archived 2026-08-03._
### 2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved
- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.**
Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the
SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo
--entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank
-w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml
--stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is
`["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash.
Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer`
(backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→
immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→
**llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→
(more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate
1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0)
but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image.
Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX,
expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.
_Archived 2026-08-03._
@@ -1,4 +0,0 @@
- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized"
is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but
fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push
finishes. FIX: re-run once built, or gate on `workflow_run: completed`.
@@ -1,4 +0,0 @@
- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools
needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The
althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO
routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`.
@@ -1,4 +0,0 @@
- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe
ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast
on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the
566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`.
@@ -1,19 +0,0 @@
- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3
epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass
`num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs
+ _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the
off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a
~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run.
FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig,
float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`.
GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min
train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step;
train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45
banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0):
ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is
COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral
literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift,
arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep
on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval +
the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO
adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).
@@ -1,8 +0,0 @@
- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.**
Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision,
util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps
ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled
refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path);
Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting
("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway
sampling-defaults wiring still PENDING.
@@ -1,11 +0,0 @@
- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD
Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.**
Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0
KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf);
(4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a
DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over
RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp
template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the
live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively
opens `<think>`/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap
b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).**
@@ -1,12 +0,0 @@
- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.**
Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path
finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied
VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before
`from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does
`model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers
5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the
DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED
un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO
confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93%
util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit.
gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.
@@ -1,17 +0,0 @@
- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.**
Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the
SFT container `aeon-t1-sft` exactly, only `--stage sft``dpo`): `sudo docker run -d --name aeon-t1-dpo
--entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank
-w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml
--stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is
`["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash.
Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer`
(backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→
immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→
**llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→
(more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate
1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0)
but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image.
Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX,
expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.
@@ -1,4 +0,0 @@
- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` +
`envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal),
YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my
parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.)
@@ -1,4 +0,0 @@
- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC
frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks-
TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the
bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real.
@@ -1,7 +0,0 @@
- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).**
Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty
idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching).
Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08).
**This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the
operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread
01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]].
@@ -1,46 +0,0 @@
- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed,
#335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer``gen` (overrides
intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on
`summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit
both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen**
(b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted
`worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL
NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1`
after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config
INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven
safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed).
StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON:
the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never
updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) —
pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass`
parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider
description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix
(operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING
defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs
when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway
config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD;
live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09):
character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving
character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping
principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the
dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent
but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44,
UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly
from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) +
mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's
`load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character +
character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window
VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev
to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked
config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly
re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard
+ character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning
downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying
(both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37**
(char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09):
mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx
131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45
canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push
of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity
block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev),
monitor armed. See [[reference_corviduo_dev_emergency_ops]].
@@ -1,4 +0,0 @@
- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8
(not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the
cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) —
that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting.
@@ -1,10 +0,0 @@
- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into
gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17,
vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack
(01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829);
max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and
`repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture
no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone
voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514).
**Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not
A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s).
@@ -1,3 +0,0 @@
- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that
spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning
(penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path).
@@ -1,16 +0,0 @@
- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker
docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB
registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the
2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to
remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because
daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never
reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd
opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`,
`rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the
frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime
(zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares
`live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start,
never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers
once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon;
log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running
containers).
@@ -1,20 +0,0 @@
- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable
systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00`
America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh`
runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in
`~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH
footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS
new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode
ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0):
web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input +
caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle
brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor);
registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired
step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default
+ `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled
1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending.
**NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start
brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session
handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 +
ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token
feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo).
@@ -1,16 +0,0 @@
- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev
requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`,
0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the
VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` =
`/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump
broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch
2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke
the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot
`/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention),
re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at
`/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):**
comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention
rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY
the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) →
rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1),
lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.
@@ -1,14 +0,0 @@
- `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1,
SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch
2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt`
FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came
only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt`
as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any
transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment).
`docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB
migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3
needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a
mmartial ComfyUI version bump, `git diff <old> <new> -- requirements.txt` — if torch is unpinned, bump the CODE
without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid
1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See
[[reference_irv_ml1_comfyui_mmartial]].
@@ -1,13 +0,0 @@
- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev
R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml`
(`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347
`session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's
personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based
`affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347.
R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe
→ present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string
insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own
yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows
baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo
untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they
want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]].
@@ -1 +0,0 @@
- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1.
@@ -1 +0,0 @@
- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot.
@@ -1 +0,0 @@
- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated).
@@ -1 +0,0 @@
- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate).
@@ -1 +0,0 @@
- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]**
@@ -1 +0,0 @@
- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize.
@@ -1 +0,0 @@
- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]].
@@ -1 +0,0 @@
- `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug.
@@ -1 +0,0 @@
- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM``model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section.
@@ -1 +0,0 @@
- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`.
@@ -1 +0,0 @@
- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528.
@@ -0,0 +1,15 @@
`[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap; fix relayed to heid, NOT applied gateway-side.**
heid reported that `kimi-k3` (the primary route = Kimi Code coding endpoint `openai/k3` @ `api.kimi.com/coding/v1`) silently degraded its cross-frontier panel: on large/reasoning-heavy dispatches, `completion_tokens: 16381` **exactly**, `content` empty, `reasoning_content` ~64KB, `finish_reason: **stop**` (a truncation mislabeled as a clean stop). `max_tokens: 100000` in the request was not honored.
**Investigation arc (a clean cross-frontier-triage + verify-on-the-wire case):**
1. My first read: a flat ~16384 OUTPUT cap; fix = a LiteLLM `stop→length` relabel callback (heid's fallback ask). Confirmed the cap isn't in our LiteLLM config (no `max_tokens` clamp on the route).
2. Operator routed a fix-research pass to **dvalin-smithy-dev + bil-smithy-dev** (independent). Both CONVERGED (docs-based): `max_tokens` is a deprecated alias on Kimi/Moonshot; the canonical field is `max_completion_tokens` (default 131072, max 1M); the coding endpoint defaults output to 16384; fix = send `max_completion_tokens` + `reasoning_effort` via `extra_body` (drop_params-safe).
3. **heid's live data REFUTED the docs hypothesis:** a later dispatch hit `completion_tokens: 18455` (ABOVE 16384) cleanly, with `reasoning_tokens: 16198` (just under 16384) and content present. So COMPLETION is uncapped; the bound is on **REASONING at ~16384**. When a hard task's thinking exhausts that budget, nothing's left for content → empty answer under `stop`.
4. **I proved it on the wire** — ran heid's real 500KB failing bundle direct at both endpoints (bypassing LiteLLM so `reasoning_effort` isn't dropped): default effort → 504/timeout (the failure); **`reasoning_effort: low` → reasoning ~1213.5k (under the ceiling), content returns (67.6k chars)**, on BOTH coding AND general endpoints. So re-routing to the general endpoint buys nothing — the fix is the effort param, and it works on the wire.
**THE FIX (caller-side, no shared-gateway change/restart):** send `reasoning_effort` via **`extra_body`** on kimi-k3 dispatches (`low` for large bundles). LiteLLM `drop_params: true` strips the top-level `reasoning_effort` — which is exactly why heid's earlier `reasoning_effort: low` was a no-op. `extra_body` survives drop_params (the house GLM-thinking pattern). Tradeoff: low effort = shallower reasoning, but a complete answer beats today's empty one.
**Relayed to heid to validate on a real round** (the one unconfirmed hop is whether `extra_body` survives OUR LiteLLM). **Backstop if it doesn't:** add `allowed_openai_params: ["reasoning_effort"]` to the `kimi-k3` route in the gateway config — that IS a shared-gateway change + a ~10s restart (blips all consumers), so it needs a heads-up.
Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/litellm/conf/config.yaml` (see Recent-decisions `[2026-07-25]` Kimi K3 wiring). No gateway change was made this session. Failing dispatch on record: `01KYTASKTY3T` (jackdaw-dev bug-hunt).
@@ -0,0 +1,24 @@
`[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev `10.250.50.152:8090`.**
WG-internal HTTP front door for the Muninn ingestion queue (`vh/muninn-gate`, muninn-dev's repo). The full provisioning ask (staging mount + closed-schema config + bearer keys + compose/WG bind) came after a 4-message discovery exchange with muninn-dev + a cross-team coordination with worldtree-dev; operator ruled the open architecture call (shared mount) and greenlit build+boot.
**Deployment (eshpfi `stacks/muninn-gate/`):**
- Image `muninn-gate:0.0.14` — no Dockerfile upstream, so infra-ops owns containerization. `python:3.11-slim` + `uv pip install .`; **`muninn-dispatch==0.1.4` from the internal Gitea index** (`[tool.uv.sources]`, `uv pip install .` honored the pin), token passed as a **BuildKit secret** (`--secret id=gitea_pw`) so it never lands in a layer. Built on corviduo-dev.
- **`ingestion_root: /data/state/ingestion`** — the `worldtree-personal_worldtree-state` docker volume mounted at `/data/state`, byte-identical to the watcher's view. **Acceptance criterion (muninn-dev's): `/health` → `watcher.running: true` PROVES byte-identity** (the gate reads the heartbeat the watcher writes); `no_heartbeat` with the watcher up = root mismatch. Verified true first boot.
- **`user: "1000:1000"`** — the ingestion dir is `vh:vh 0755`, so a non-root gate had to run as uid 1000 to WRITE the queue (my Dockerfile's `USER gate`/10014 would've been denied; the watcher itself runs as root and bypasses perms). This uid requirement was a genuine spec gap — muninn-dev added it to the contract (`084526e`, vh:vh 0755 + 1000:1000 as the worked example) so no future deployer re-derives it. `ingestion_root_writable: true` in `/health` is the post-deploy confirmation.
- **staging `/mnt/muninn-staging/mimir-inbox`** — bound `:ro`, SAME absolute path in BOTH the gate AND the watcher (dispatch stores paths absolutely; the watcher opens them at claim time). worldtree-dev added the watcher-side bind (their image) in **b162** (`${MUNINN_STAGING_DIR:-…}:/mnt/muninn-staging/mimir-inbox:ro`). Currently a **LOCAL placeholder dir** on corviduo-dev.
- config (single-writer) `/opt/docker/conf/muninn-gate/muninn-gate.yaml` (0600, 1000:1000). Schema CLOSED (unknown field = boot failure). 2 bearer keys minted: `mimir-inbox` [read,submit], `ops-curl` [read,submit,control]. `network_mode: host`; health probe = **`/ping`** (NOT `/health`, which is always-200 by design and would never restart the gate). Committed `786462a` (no secrets).
**Verified boot:** `/ping` `{"service":"ok"}`; `/health` (ops-curl bearer, 200) `watcher.running:true` + `ingestion_root_writable:true`. muninn-dev independently poked the live gate — auth/route surface all held (401s w/ `WWW-Authenticate: Bearer`, the 4 FastAPI default routes gone, error-envelope-not-307 on trailing slashes = bug-hunt findings 5+6 confirmed outside pytest).
**DEFERRED (the submit path) — the mimir-inbox era:** SUBMIT returns `not_found` against the placeholder staging (correct, not a defect — muninn-dev confirmed) until the real staging dir + a mimir-inbox writer exist. ~~Operator ruled shared mount (mimir-inbox stays off-box, writes to a shared/NFS mount both gate + watcher bind at the same path).~~ **SUPERSEDED 2026-08-01 — operator REVERSED to CO-LOCATE:** mimir-inbox runs ON corviduo-dev, alongside the gate + watcher, staging = a corviduo-dev-LOCAL dir (not NFS). Reason the off-box/NFS call fell: muninn-dev's code-check showed staging is NOT same-fs-constrained (gate reads staging metadata + passes path strings; `os.replace` is inside `ingestion_root`), so staging's real constraint is **path identity across writer/gate/watcher**, which co-location buys outright — and it sidesteps the NFS failure modes (path-identity break, TOCTOU widening, stale handles, a hung mount blocking `resolve(strict=True)` — the last of which blocks mimir-inbox's *event loop*, not just a threadpool worker, since its staging check is in an async handler). Ruling relayed 3× (muninn-dev ×2 w/ msg-id citations, mimir-dev ×2) + operator in-session; **mimir-inbox key handed over 2026-08-01** (bumped to [read,submit,control], 0600 drop on nh3-dev). Tail on the co-locate ruling: raise worldtree-dev (box-side provisioning + the watcher claim-semantics open Q) → provision the real corviduo-dev-local `/mnt/muninn-staging/mimir-inbox` (uid = mimir-inbox's runtime identity, rw-writer / ro-gate+watcher) → 0600 key drop on corviduo-dev → muninn-dev's **one-file path-agreement probe** → acceptance. NB gate submit surface = **`POST /jobs`** (path-addressed; NO `POST /upload` — upload deferred v0, gate never ingests bytes). `staging_roots` already allowlists the path (no gate-config change).
**RESOLVED 2026-08-01 (worldtree-dev, from source `core/muninn/runner.py:362-367`):** the watcher **OPENS the staged file in place** at claim (`parse_document(file_path)` on the dispatch-recorded absolute path) — it never moves/copies the source into the job dir (job dir holds DERIVED artifacts only). Consequences: (1) staging needs **PATH IDENTITY only**, so **co-location is a CONVENIENCE, not a requirement** — the parked multi-host option stays fully viable with a shared mount at the same absolute path on both hosts. (2) The real same-fs constraint is `.enqueue-tmp/``os.replace` into `pending/`, same-fs with `ingestion_root` — never staging (confirms muninn-dev). (3) **⚠️ OPERATIONAL RULE for mimir-inbox lifecycle (worldtree-dev):** open-in-place means the staged file MUST stay present+readable from submit **until the job is TERMINAL** (complete / failed-and-not-retried) — retry re-runs the structure phase, which re-opens the staged path. A cleanup that deletes on 201-submit kills every job at claim with a not-found that looks EXACTLY like the namespace-mismatch failure the bind exists to prevent. Relayed to mimir-dev for their cleanup design. **Gate-side edge (muninn-dev):** `POST /jobs/{id}/retry` returns `200 {requeued}` even for a job whose staged source was deleted — `muninn_dispatch.requeue` validates job STATE not file existence, and admission isn't re-run on retry (nothing re-stats files) → a FALSE success that dies at claim. Gate deliberately unguarded (re-admit re-resolves under a new clock, still races; lifecycle is the writer's), recorded as a gate compatibility constraint. So the retention rule isn't just "avoid claim-fail" — it's "retry will LIE with a 200 if the source is gone."
**worldtree-dev approved co-location** (2026-08-01): another small infra-ops-managed LAN/WG-internal service on corviduo-dev in the gate's posture is fine at their OS/app layer; port/supervision/identity mine to shape; staging-dir ownership flip (mimir-inbox-writable, gate+watcher :ro — b162 watcher bind already :ro) at my convenience. **NEXT: coordinate the mimir-inbox deploy inputs with mimir-dev** (image/build recipe — likely infra-ops containerizes like muninn-gate; app config/env; port), then provision staging dir + stand up the service (uid 1000, matching the corviduo-dev muninn stack) + 0600 key drop on corviduo-dev + muninn-dev's path-agreement probe + acceptance.
**Operational guard (no auto-check exists):** docker fabricates a MISSING bind source as an empty dir that passes every closed-config check → **confirm the host mount actually exists before wiring/repointing a bind** (`os.path.ismount` breaks on subdir roots; emptiness is normal pre-first-upload). This is why the gate/watcher path-agreement is an operational discipline, not a validated invariant.
**Hardening candidate (flagged, not done):** the compose mounts the WHOLE `worldtree-personal_worldtree-state` volume at `/data/state` per muninn-dev's spec; a subpath mount of just `ingestion``/data/state/ingestion` would be tighter (gate only needs RW on ingestion). Confirm with muninn-dev before adopting.
See auto-memory `reference_muninn_gate_deploy`, `reference_muninn_gate_staging_path`; [[2026-07-25-infra-ops-wt-config-repo]] (corviduo-dev boundary), and Recent-decisions `[2026-07-27]` muninn watcher sidecar (the other half of #377).
@@ -0,0 +1,32 @@
`[2026-08-02]` **The mimir-inbox / #377-read-path arc — deploy, four bugs found+fixed+verified, a cloned voice, all in one long session (2026-08-01→02).**
The browser-facing half of the #377 Muninn ingestion arc, end to end: mimir-inbox stood up, the write path proven, the read path chased through four defects to a verified-working state, and a character voice cloned into the TTS zoo. Peers: mimir-dev (the app), muninn-dev (gate/watcher spec), worldtree-dev (Worldtree app layer + the #380/#381/#382/#383 fixes), ratatoskr-dev (a consumer + the rigorous verifier).
## mimir-inbox deployed (#377)
- **New infra-ops stack, canonical eshpfi `stacks/mimir-inbox/`; live corviduo-dev `10.250.50.152:8091`** (co-located w/ muninn-gate :8090 + the worldtree-personal muninn watcher). Full deploy detail + procedures → auto-memory `reference_mimir_inbox_deploy`.
- **Placement decision (operator, reversed):** 7-31 he ruled mimir-inbox stays OFF corviduo-dev (shared/NFS mount); 8-01 he REVERSED to CO-LOCATE. Trigger: muninn-dev's code-check showed staging is NOT same-fs-constrained (gate reads staging metadata + passes path strings; `os.replace` is inside `ingestion_root`) — staging's real constraint is **path-identity across writer/gate/watcher**, which co-location buys outright while dodging NFS failure modes. I HELD the reversal for the operator's direct word (data/hosting on a team-managed box, reversing his own ruling) even against 3 peer relays — vindicated as the right instinct; muninn-dev agreed.
- Build: **`uv sync --no-dev --frozen`, SINGLE-STAGE** (project installs editable-linked to `src/`, so src/ MUST stay beside .venv — a multi-stage "copy only .venv" dies at import/404s assets). uid 1000, host-net bind 10.250.50.152:8091, TCP-liveness healthcheck (deliberately NOT gate-coupled). Redeploy = refresh build context (**preserve the on-server `.env`!**) → `docker build -t mimir-inbox:0.0.1 -t mimir-inbox:<sha> .``compose up -d`. Version stays 0.0.1 across dev commits → tag the image w/ the source SHA too. Live commit progression `0478452``c8ab38f``2dcc77e`→**`8ece117`** (3 redeploys).
- mimir-inbox key on the gate bumped [read,submit]→**[read,submit,control]** (cancel/retry); brokered via a 0600 drop on nh3-dev (never on the althing bus).
## The read-path bug chain (worldtree-dev's, all found via this arc)
- **#380 wing-blind indexing:** the book-ingest path upserted concepts into a hardcoded `main` Chroma collection while wing search reads the `fiction` collection → P&P written to disk but `search_library` returned total 0. A silent-success defect ("complete/69 indexed" was right about the WRITE, wrong drawer). Root-caused off MY physical evidence (files on disk + search empty). Fixed b164 + a one-shot `--reindex <job_id>` (re-upsert into the right wing collection + delete stray `main` rows).
- **#381 stale Chroma client:** the personal api opens its Chroma client before the watcher's cross-process writes → **a freshly-ingested/re-indexed book is NOT queryable until the api is restarted.** Proven by my restart-diagnostic (pre-restart total 0 → post-restart hits, same index). Workaround until fixed: `docker restart worldtree-personal-worldtree-api-1` after any ingest/re-index. Filed as #381.
- **#382 unreliable Mimir grounding (the subtle one):** post-#380-fix the index was correct, but Mimir's grounding was INTERMITTENT — some sessions navigated the opaque job-hash dir (`mimir-f3887c9b97b7`) to the content, others distrusted the correct vector hits and **silently answered from training knowledge** (worst of the looks-fine-isn't family). ratatoskr-dev caught it; I'd been over-confident ("Mimir read Austen back to you") having verified the INDEX, not the GROUNDING. Fixed b166 with BOTH shapes: a self-describing `_index.md` per wing job-dir (resolves the hash dir to its title) + a Mimir prompt rule (wing-scoped hits ARE library content, never discard on a name mismatch, never substitute training). **Verified: ratatoskr-dev re-ran 3× fresh sessions → 3/3 grounded**, citations in note-extracted language not raw Austen. #382 CLOSED.
- **DCC (Dungeon Crawler Carl, job `b59c147c5ce0`) backfill:** `--reindex` FAILED ("job not found in any state dir" — predates state-tracking). SETTLED = **no re-file** (the b166 prompt rule already grounds it even without an `_index.md`; ratatoskr confirmed incidentally); an `_index.md` rides whenever DCC is next re-ingested.
- **#377 mimir-inbox banner bug (mimir-dev's, `8ece117`):** `/health-banner` misattributed an unwritable `ingestion_root` to the WORKER, rendering "The worker is not running." for a running worker — a false lead pointed at infra-ops's half of #377. Fixed (guard split into two banners); I confirmed from the DEPLOYED handler (not just the test) that `ingestion_root_writable:False` now renders "The ingestion root is not writable."
## muninn-gate → muninn-dispatch 0.1.5
Rebuilt `muninn-gate` off `vh/muninn-gate` main `bc04c4c` (dispatch 0.1.4→0.1.5) so the gate serves the new `concept_schema`/`concept_schema_source` row fields (computed gate-side). Gate version unchanged 0.0.14 (dual-tag the SHA). Build needs the vh gitea token as a BuildKit secret (`--secret id=gitea_pw`, UV_INDEX_GITEA_USERNAME=vh, drop+shred). Recreate with `compose up -d` (NOT bare restart — needs the new image). Verified: P&P job serves `concept_schema='fiction'`, `concept_schema_source=null` (null correct — pre-b164 job). Registry tags by commit SHA — `v1.0.0bNNN` docker tags don't resolve; use the deployed SHA (confirm `--reindex` present before using an image for a data-op).
## donut voice (65-frost → Zonos gateway)
Operator: "pick up 65-frost, use that bundle as a voice for a character named donut." 65-frost = a **Booth id** (`~/booth-data/65-frost/`) holding a curated yt-voice-clipper dataset (`dataset-…-curated.zip`: 4 clips + manifest, all SPEAKER_02 = Princess Donut). **Zonos gateway voice registry = a filesystem drop:** `<Name>.wav` in the voices dir (44.1kHz mono s16 PCM) auto-registers as `voice:"<name>"` on **startup** (needs a restart). The LIVE dir is the bind mount `/opt/docker/compose/zonos-gateway/voices/` (lkraven-writable), NOT the working tree. Built `Donut.wav` from seg000 (best clip), dropped it, restarted → `voice:"donut"` live in the gateway AND the Asset Engine's make form. Also copied to the build-source tree `~/zonos-gateway/voices/` for rebuild-durability (true canonical = the gitea repo, not yet CI-wired). Auditioned in booth `donut-voice`. **Expanded 2026-08-02 (onyx-58 bundle):** operator curated a 2nd Booth bundle `onyx-58` (`dataset-467d2cf8…curated.zip`, 3 Donut clips) as additions. Rebuilt the reference = **seg000 (65-frost) + seg101/seg110/seg148 (onyx-58)** ffmpeg-concat + resampled 24k→44.1k mono s16 = **52.0s**. `seg148` was diarized SPEAKER_03 but is Donut (operator-confirmed misdiarize → included). Assembly is NOT `assemble_voice.py` (that `-c copy` can't resample + caps ~15s); used a manual `aresample=44100,aformat=…,concat=n=4` filter. Backed up old ref → `irv-ml1:~/Donut.wav.pre-onyx58`; dropped to live bind-mount + build-source tree; `docker compose restart` (healthy 2s, `voice:"donut"` still 1 of 9). A/B booth `donut-onyx58` (A=old 16.3s ref, B=new 52s ref, same line). Longer ref is fine mechanically: gateway passes it as `speaker_audio_base64` → speaker *embedding*, not an audio prefix. **BUT auditioned → REVERTED same day:** pinned-seed neutral A/B (5 pairs, booth `donut-onyx58`) showed the single-clip seg000 (16.3s) beats the 52s 4-take concat on timbre — concatenating disparate takes muddied the embedding more than the range helped. Reverted both live + build-source to seg000-alone. Lessons (→ Tried-and-abandoned): more reference ≠ better when takes vary; and **emotion steering pulls output away from the clone fast** (operator craft rule) — keep clones emotion-neutral; bare `{input,voice}` calls send NO emotion (gateway only enables it on an explicit `emotion_*`/`preset` dial).
## Zonos streaming (no gateway change needed)
ratatoskr wanted play-as-it-arrives. `/v1/audio/speech` ALREADY streams — chunked `StreamingResponse`, opens native `/tts/generate` with `stream=True`, wraps as a streaming int16 WAV with `0xFFFFFFFF` placeholder sizes (meant for progressive `<audio>`). Verified TTFB 0.44s vs 6.84s total, `transfer-encoding: chunked`, dials preserved. ratatoskr's proxy was rewriting the placeholder header → forced buffering. Fix was theirs (pass chunks through); shipped + confirmed (TTFB 0.46s progressive). The Asset Engine (ana-docker:8200) IS the fleet "TTS zoo" (~20 audio svcs w/ irv-ml1 endpoints); zonos-gateway registered there, state=ready.
## Lessons (also in Tried-and-abandoned)
- **Verifying the INDEX (search returns hits) is NOT verifying GROUNDING** (does the agent trust+use them vs. silently answer from training). Check that citations are note-extracted, not model-knowledge. ratatoskr caught this after my over-confident "it works."
- **Reading the DEPLOYED artifact > trusting the test** for "is the fix live" — the test proves the source is right; reading the running code proves the artifact is, which is what an on-call actually meets.
- Held a boundary-box/data reversal for the operator's DIRECT word against 3 peer relays — the right call (peer relay ≠ operator consent; the placement guard was vindicated).
See also: [[2026-07-31-muninn-gate-deploy]]. auto-memory: `reference_mimir_inbox_deploy`, `reference_muninn_gate_deploy`, `reference_muninn_gate_staging_path`, `reference_zonos_tts_stack`, `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
@@ -0,0 +1,17 @@
**Worldtree b168/#384/#385 arc — COMPLETE 2026-08-03.** A long peer-driven arc across worldtree-dev / muninn-dev / mimir-dev / ratatoskr-dev, all on corviduo-dev's demo+personal instances. Sequence: providers.yaml boot-gate pre-sync → b168 deploy → DCC #384 reindex → round-2 full re-ingest → #381 restart → operator-approved production dedup sweep. Landed clean; three of MY foot-guns along the way, each caught + hardened into a fleet runbook rule (see Tried-and-abandoned: `mv -t`, `docker exec -u 1000`, shared-containerd race).
## providers.yaml pre-sync (boot-gating config)
b168 (commit `293f8f3`) added a `summarization` capability block that in-image `agents/muninn/config.yaml` references → boot-blocking if the host bind-mounted providers.yaml lacks it. Synced both hunks (summarization block + deep-reasoning desc) into demo+personal via `deploy-wt-config`; instance-configs commit `53349f8`.
- **deploy-wt-config runbook:** `~/development/worldtree-instance-configs/scripts/deploy-wt-config {diff|deploy|capture} <inst> --file providers.yaml` (per-instance dirs demo/personal/pinned; `deploy` = host write + api/matrix restart + 90s health-gate + auto-rollback; `diff`/`capture` safe). demo+personal providers.yaml are byte-identical.
- **GOTCHAS:** (1) an UNPUSHED source commit → `git show <sha>` 404s and a gitea `raw?ref=<sha>` silently falls back to the default branch; verify the commit exists (`/git/commits/<sha>`) before trusting a fetch, else ask the peer to paste hunks. (2) a peer's hunk paste may be mis-indented (8-space vs the block's 4-space) → invalid YAML; always YAML-validate after a paste-sourced edit.
- **Config-delta pre-sync rule (verified via `docker inspect`):** worldtree containers bind-mount ONLY `config/` host-side (`/opt/worldtree-*/config/` → providers/model_roles/matrix/policies/defaults/env.public = the pre-syncable set); `agents/` (schemas.yaml, prompts) + all code ship IN-IMAGE. So only a `config/*.yaml` change is boot-blocking-pre-syncable; an `agents/`-or-code delta needs NO host pre-sync (CI carries it). b169's schemas.yaml (#387) was correctly no-pre-sync.
## #384 reindex + #381 restart + verify
DCC job `mimir-6351554e8e8f`. Reindex: `sudo docker exec -u 1000 worldtree-personal-worldtree-muninn-1 python -m core.muninn --reindex <job>` (⚠️ MUST `-u 1000` — default-root writes contaminate the uid-1000 KB tree; see Tried-and-abandoned). Then **#381 restart** (stale-Chroma-client fix): `sudo docker restart worldtree-personal-worldtree-api-1` (plain bounce, NO compose up / no image repoint) → healthz/readyz 200 ~25s.
- **Chroma-verify runbook:** `sudo docker exec -i <muninn> python -` (MUST pass `-i` or stdin never reaches `python -`) → `chromadb.PersistentClient('/data/kb/.chroma').get_collection('fiction').get(where={'job_id':<job>}, include=['metadatas'])`. Chroma persists at container `/data/kb/.chroma` = host volume `worldtree-personal_worldtree-kb`.
- **Retrieval-visibility check (NOT grounding — that's ratatoskr's):** a Mimir session — admin token `~/.config/worldtree/personal-admin-token` (wildcard scope) → POST `/sessions` (agent_id=`mimir`, `record_tool_intermediates=true`) → POST `/sessions/{id}/messages` (STREAMS SSE, not JSON) → parse SSE `tool_result` for `search_library` wing hits → DELETE session.
## Production dedup sweep (operator-approved)
Deleted the 785 April-era DCC orphan rows (`job_id=b59c147c5ce0`, no wing/source_identity metadata → predate identity tracking) from the `main` collection. Supervised protocol: read-only verify count == 785, back up all rows (ids+docs+embeddings) to `corviduo-dev:/tmp/main-sweep-backup-b59c147c5ce0.json` (reversible), `main.delete(where={job_id})` (assert target==785 first), verify `main` 4009→3224, then **bounce the api** (a separate-process delete leaves the api's in-memory HNSW index holding the vectors until reload — the #381 pattern generalizes to deletes), confirm search now fiction-only. Backup left for /tmp natural cleanup (fiction wing is canonical; `~/archives` has the historical record).
Result: fiction wing 166 → 1,372 concepts; three consumer verify rounds 0/5 → 5/5 → saturated; #385 budget fix validated (705 vs April's 785 control, extraction AND indexing, zero truncations). worldtree-dev filed #388 for a deploy concurrency-lock (the shared-containerd race fix). See [[2026-08-02-mimir-inbox-arc]].
+52 -69
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-07-25_
_Last updated: 2026-08-03_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -109,25 +109,53 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-07-25 — **DONE: infra-ops Worldtree config-as-code repo BUILT + PUSHED** (`vh/worldtree-instance-configs`, private, gitea). Seeded byte-exact from live demo+personal `/opt/<instance>/config` state; `scripts/deploy-wt-config` (diff / deploy / capture) with backup + health-gate + auto-rollback; all three verbs live-validated (both instances byte-in-sync, capture round-trips clean, pinned refused). Local clone `~/development/worldtree-instance-configs`. **pinned confirmed OUT-OF-SCOPE** — no `/app/config` bind-mount, config baked into frozen image `446e5807` (2026-05-13). Deploy restarts api+matrix (matrix shares the config mount, no healthcheck), gates on api `/health`. **BOUNDARY AGREED** — worldtree-dev consented (althing `01KYCAECRW…`, 2026-07-25): no hand-edits to `/opt/<instance>/config`; config changes route to infra-ops as deltas (they own CONTENT + approval trail, infra-ops lands+deploys — the wyrd-grant shape). Three-layer model they hold: image `config/` = BASELINE new instances seed from (theirs) → `vh/worldtree-instance-configs` = per-instance truth (ours) → host bind-mount = deploy target (written ONLY by the tool). **Carve-out:** worldtree-dev's admin-API ops (`/admin/keys` mint, tier changes, session retirement, future runtime-grant surfaces) mutate instance DATABASES not config files → NOT config edits, stay in-band (the tool only writes config bind-mounts, never DBs). b132 CONFIG BASELINE breadcrumb composes (its INFO line = config-as-code diverges from image baseline, by design). **No live deploy done/needed** (repo already == live). Gitea create used vh creds one-time (operator-authorized) via ana-docker localhost API; pushed over internal git-SSH `10.250.50.70:222`. Full shape → `persistent-memory.d/2026-07-25-infra-ops-wt-config-repo.md`. **Earlier this session:** The Booth shipped (v0.1.3, `services/booth/`, nh3-dev :8090, Homepage-linked, upload-pickup + image-viewer + copy-id); jackdaw-compose backend deployed (nh3-dev :8787) + its throwaway cloudflare tunnel torn down; ana-ml2 README refreshed to live GPU state; Worldtree #376 arc CLOSED (per-instance config ruled by-design; wyrd grant live; drift-watcher built+retired) → `persistent-memory.d/2026-07-25-wt-376-per-instance-config-arc.md`._
_As of 2026-08-03 — **session at a natural close; nothing infra-ops-side blocked.** Two long arcs landed: (1) the mimir-inbox / #377-read-path arc (2026-08-01→02); (2) the **worldtree b168/#384/#385 arc COMPLETE** (2026-08-03 — providers.yaml pre-sync → deploy → DCC+P&P re-ingest 705+667 concepts → #381 restart → operator-approved production dedup sweep; full detail in the 2026-08-03 Recent-decisions entry + `persistent-memory.d/2026-08-02-mimir-inbox-arc.md`). Only live watch = worldtree-dev re-running the personal **b169** deploy (all-clear given). Headlines:_
- **mimir-inbox DEPLOYED + verified on corviduo-dev `10.250.50.152:8091`** (#377 browser-facing half; write+read proven end to end). Live commit `8ece117` (3 redeploys); co-located per the operator's reversed-to-CO-LOCATE ruling. auto-memory `reference_mimir_inbox_deploy`.
- **#377 read path fully working** — P&P ingested + queryable via Mimir on personal :8081. Chased through worldtree-dev bugs **#380** (wing-blind index → concepts in wrong collection; fixed b164 + one-shot `--reindex <job_id>`), **#381** (stale Chroma client → **restart `worldtree-personal-worldtree-api-1` after any ingest/re-index** until their fix), **#382** (intermittent Mimir grounding / silent training-substitution; fixed b166 = `_index.md` per wing job-dir + a prompt rule; **verified 3/3** by ratatoskr-dev). DCC re-file SETTLED = **no** (prompt rule grounds it; `_index.md` rides next re-ingest).
- **muninn-gate → muninn-dispatch 0.1.5** (rebuilt off `vh/muninn-gate` `bc04c4c`, image 0.0.14; serves `concept_schema`/`concept_schema_source`). BuildKit gitea secret; recreate w/ `compose up -d` not bare restart.
- **donut voice** cloned from the **65-frost Booth bundle** → registered in the Zonos gateway (`voice:"donut"`, live in the Asset Engine TTS-zoo make form; auditioned in booth `donut-voice`). **onyx-58 expansion TRIED → REVERTED 2026-08-02:** folded the `onyx-58` bundle (seg101/seg110/seg148, all Donut — `seg148` was diarized SPEAKER_03 but operator-confirmed misdiarize) in alongside seg000 → 52.0s multi-clip ref, but a pinned-seed **neutral/no-emotion** A/B (5 pairs, booth `donut-onyx58`) showed the **single-clip seg000 (16.3s) wins on timbre fidelity** — the 4-take concat muddied the speaker embedding. **LIVE = seg000-alone** (reverted both live bind-mount + build-source tree; old ref was at `irv-ml1:~/Donut.wav.pre-onyx58`). See Tried-and-abandoned for the emotion-fidelity lesson. Zonos `/v1/audio/speech` already streams (chunked, TTFB ~0.44s) — ratatoskr shipped the client-side chunk-passthrough for play-as-it-arrives; **no gateway change was needed**.
**Open follow-ups (non-blocking — pick one up or not):**
- **Zonos emotion:** sad axes/text pass on the 3 calibrated voices (only named-sad, untested); emotion-congruent-text pass (validates intensity, may rescue sad id); clone-char (Emmie/Penny/Natalie/Miranda) emotion rows use the mid-region fallback until measured. Presets are **provisional** (neutral-text ear-check was inconclusive). Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio,assemble_voice}.py` (run ON irv-ml1; dial-in studio = nohup :8898 on nh3-dev). dvalin thread at rest (`01KXT12FN0AS…`). → `persistent-memory.d/2026-07-18-zonos-gateway-0.2.1-emotion-presets.md`
- **zonos-gateway CI-wire:** deployed irv-ml1 tree `/opt/docker/compose/zonos-gateway` is still NON-git (hand-updated build context) — git-connect + build-on-push like the other sisters. (Same pattern soong-lab now has.)
- **soong-lab:** cutover DONE + **auto-redeploy DONE + validated 2026-07-18** (CI-deploy step live; dispatch run #5 recreated the live container ...541f7730 → ...07526a08, health-gated green). Deploy dir now **/opt/soong-lab** (deploy-owned, mirrors /opt/worldtree); old `/home/infra-ops/soong-lab-deploy` retired (`.retired-20260718`). Dedicated soong-only ed25519 deploy key on `deploy`'s authorized_keys (fp SHA256:MG7M3Ri…). → `persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md`.
**Zonos voice stack (LIVE, unchanged):** 8 voices in `zonos-gateway` (`:8890` irv-ml1) — defaults AmericanFemale/Male/BritishFemale/Cora + 4 clones Emmie/Penny/Natalie/Miranda; add a voice = drop `<Name>.wav` in `/opt/docker/compose/zonos-gateway/voices/` + `docker compose restart` (host-managed bind-mount, NO rebuild). Clone pipeline: `/mnt/smithy/voice_clones/<name>.zip``assemble_voice.py` → drop. Dial-in studio http://10.100.10.50:8898/ (nohup on nh3-dev, relaunch `nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &`).
**Zonos voice stack (LIVE):** **9 voices** in `zonos-gateway` (`:8890` irv-ml1) — defaults AmericanFemale/Male/BritishFemale/Cora + 4 clones Emmie/Penny/Natalie/Miranda + **donut** (2026-08-02, from the 65-frost bundle, **expanded same day with onyx-58 clips → 52.0s multi-clip ref**); add a voice = drop `<Name>.wav` (44.1kHz mono s16 PCM) in `/opt/docker/compose/zonos-gateway/voices/` + `docker compose restart` (host-managed bind-mount, NO rebuild; registry scans at startup). Also mirror into the build-source tree `~/zonos-gateway/voices/` for rebuild-durability. Clone pipeline: `/mnt/smithy/voice_clones/<name>.zip``assemble_voice.py` → drop. Dial-in studio http://10.100.10.50:8898/ (nohup on nh3-dev, relaunch `nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &`).
**althing monitor** ARMED (handle `infra-ops`, wake-listener `bwraz1ai5`; herald up). ⚠️ Re-arm ONLY after an actual FIRE (`<task-notification> completed rc0`), never after a plain operator turn — re-arming while the prior listener is still live bounces rc3, and chaining the arm with `&`/`&&` orphans it (untracked → mail unwatched). Spawn `althing-wake-listener` as its OWN run_in_background task. Open watch: worldtree-dev (#376 closed; #363 research-wing ingest PARKED).
**althing monitor** ARMED (handle `infra-ops`; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after an actual FIRE (`<task-notification> completed rc0`), never after a plain operator turn (bounces rc3). Spawn `althing-wake-listener` as its OWN `run_in_background` task — NEVER chain with `&`/`&&` (orphans it untracked → mail unwatched). **ACTIVE WATCH — b169 personal deploy:** worldtree-dev re-running the staging/v1.0.0b169 personal deploy after a pull-fail I diagnosed as a transient shared-containerd concurrent-pull race (NOT disk); all-clear given = just re-run, **do NOT prune** (`6e34a87` is in-use by the running demo instance — pruning would down demo; see Tried-and-abandoned). worldtree-dev verifies health + filed #388 for a deploy concurrency-lock. **PARKED watches:** #363 research-wing ingest (no deadline); worldtree-dev's `/embed` fix reaching PERSONAL (503 on b146 until a staging promotion — operator's call). The **worldtree b168/#384/#385 arc is COMPLETE** — runbooks + full detail in the 2026-08-03 Recent-decisions entry.
**eshpfi UNPUSHED** — the Booth arc (`f4a5ba7`..`d813f15`), ana-ml2 refresh (`966324c`), nh3-dev inventory (`cd4d52e`) + this snapshot are committed but unpushed (push = operator's call). `stacks/heretic2-charrp-reasoning/` UNTRACKED; `graphify-out/GRAPH_REPORT.md` modified.
**Two small pending items (operator's call, non-urgent):** (1) bless/reshape the `env.public` non-secret-env-overlay mechanism in the config repo; (2) the pre-existing herald pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range" — likely `render_command messages[0]` on empty list; NOT infra-ops's, rec = flag to althing-dev).
**eshpfi has UNPUSHED local commits**`main` is **~21 ahead of `origin/main`** (mostly 2026-08-02→03 memory commits from the worldtree arc + this snapshot; plus the earlier mimir-inbox/muninn-gate/Audio8/#383 work). **Push is the operator's call.** `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's); `graphify-out/GRAPH_REPORT.md` = graphify-hook artifact (churns on every commit, ignore, don't stage).
**PARKED (grok-code/Codex):** operator asked about fronting grok-code / Codex behind the LiteLLM gateway. Rec (given): raw models behind the gateway → **API keys** (native `xai/` + `openai/` providers, the GLM-passthrough pattern); fleet *consults* → the **Heid/Eitri peer-CLI** pattern (Codex already wired). Do NOT reverse-proxy the subscription CLIs (grok CLI / Codex CLI, OAuth-auth) into the gateway — ToS + account-ban risk + brittle. Untracked by operator choice; no decision made.
**Carried standing (non-blocking):** ana-ml2 GPU0 ~14 G reserve; irv-ml1 3090 oversubscription (kokoro :8193 + vibevoicefusion :9527 idle-pinned + zonos :1920 — operator declined to fix); rotate the 5 rest-server backup creds (operator, offline); Worldtree #363 research-wing ingest (parked, no deadline); T1 SFT LoRA dormant; Zonos2 engine still NATIVE (containerize deprioritized).
**Carried standing (non-blocking):** ana-ml2 GPU0 ~14 G reserve; irv-ml1 3090 oversubscription (kokoro :8193 + vibevoicefusion :9527 idle-pinned + zonos :1920 — operator declined to fix); rotate the 5 rest-server backup creds (operator, offline); Worldtree #363 research-wing ingest (parked, no deadline); T1 SFT LoRA dormant; Zonos2 engine still NATIVE (containerize deprioritized); **Audio8 → TTS zoo** (PARKED 2026-08-02 per operator — download Audio8 + add it to the fleet TTS zoo [= the Asset Engine catalog at ana-docker:8200, ~20 audio svcs w/ irv-ml1 endpoints; Zonos native :1920, zonos-gateway :8890, chatterbox-fast :8197, etc.] DEFERRED, not now; un-park = confirm Audio8's source/nature w/ operator, then register as a new Asset-Engine service entry [endpoint on irv-ml1] per the zoo convention); **worldtree config-fold PENDING** (next routine config sync): add `reference_knowledge.tier3_wings: ["fiction"]` to BOTH demo+personal defaults.yaml (worldtree #383, b5db691, PARITY-ONLY — baked default matches so no behavior diff, not boot-blocking; per-instance widening [e.g. +main on personal] is this key's purpose).
## Recent decisions
- `[2026-08-03]` **worldtree b168/#384/#385 arc COMPLETE** — providers.yaml boot-gate pre-sync → b168 deploy → DCC+P&P re-ingest (705+667 concepts, 0 truncations, #385 budget fix validated vs April's 785 control) → #381 restart → operator-approved production dedup sweep (785 April orphans deleted from `main`, 4009→3224). Fiction wing 166→1,372 concepts; consumer verify 0/5→5/5→saturated. Three of MY foot-guns hardened into fleet runbook rules (`mv -t`, `docker exec -u 1000`, shared-containerd pull-race — see Tried-and-abandoned). Full runbooks (deploy-wt-config, Chroma-verify, config-delta pre-sync rule, #381, sweep) → `persistent-memory.d/2026-08-03-worldtree-b168-384-385-arc.md`
- `[2026-08-02]` **mimir-inbox / #377-read-path arc — deployed + 4 bugs found/fixed/verified + a cloned voice.** mimir-inbox live on corviduo-dev:8091 (#377 write+read proven, live `8ece117`); worldtree-dev #380 (wing-blind index) + #381 (stale-client restart) + #382 (intermittent Mimir grounding) chased and **verified 3/3** by ratatoskr-dev; muninn-gate → dispatch 0.1.5; **donut** voice cloned from the 65-frost Booth bundle into the Zonos gateway; Zonos streaming confirmed already-working. Full arc, procedures, and lessons → `persistent-memory.d/2026-08-02-mimir-inbox-arc.md`
- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md`
- `[2026-07-31]` **worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred.** memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on `vh/worldtree-sdk` (source pulled via the **Gitea API archive** — git-HTTP 403s on that repo); publishing to the vh USER namespace **can't be delegated** (401 `reqPackageAccess` even with `write:package`) so it needs an owner token — operator saved a **FULL vh site-admin token at `~/.config/gitea/vh-token` (0600)** for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
- `[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap.** heid's cross-frontier panel was silently degraded (empty content, `finish_reason: stop`). dvalin+bil researched (docs said deprecated-max_tokens); heid's live data refuted that (completion hit 18455) → it's a reasoning ceiling. **Proven on the wire against heid's real 500KB bundle:** `reasoning_effort: low` drops reasoning under the ceiling → content returns, on BOTH coding + general endpoints. Fix is CALLER-side (no gateway change): send `reasoning_effort` via **`extra_body`** (LiteLLM `drop_params: true` strips the top-level param — why heid's earlier attempt no-op'd). Relayed to heid to validate; backstop = `allowed_openai_params` on the route. → `persistent-memory.d/2026-07-31-kimi-k3-reasoning-cap.md`
- `[2026-07-27]` **Zed edit-predictions: keyless FIM-completion route SHIPPED end-to-end.** Operator wants Zed's inline edit-prediction (which CANNOT send an auth header) to reach a FIM coder via `/v1/completions`. **Deep-research (106-agent workflow) picked `Qwen/Qwen2.5-Coder-1.5B`** (BASE, Apache-2.0; native FIM `<|fim_prefix|>/<|fim_suffix|>/<|fim_middle|>` IDs 151659/60/61; Zed `prompt_format:"qwen"`). Runner-up 3B = non-commercial Qwen-Research license; **no small dense Qwen3-Coder exists (all MoE, smallest 30B)**. **Stood up `vllm-coder`** on ana-ml2 **GPU1 :8020** (served-name `qwen2.5-coder-1.5b`, 8192 ctx, util 0.06, fp8 KV). To fit, **shrank granite (phasing out, operator-directed):** util 0.27→0.13, max-len 131072→16384, seqs 1024→256 (freed ~14 GB; the KV-≥-1×-max-len rule crash-looped it at util 0.12/32768 → settled 0.13/16384). **LiteLLM alias `coder-fast`** → :8020 (`mode: completion`). **Minted a `coder-fast`-SCOPED virtual key** (verified 403 on `gen` — the real blast-radius bound). **Built `zed-fim-proxy`** (ana-docker **:4141**, `network_mode: host`, stdlib-python, `stacks/zed-fim-proxy`): keyless POST `/v1/completions`, model-allowlist `coder-fast`, injects the scoped key → LiteLLM :4000; `GET /ping` anon liveness; wrong-model→403, wrong-path→404, `/chat/completions` rejected. Verified keyless FIM end-to-end ('a + b', finish `stop`). **Zed `api_url` = `http://10.250.50.70:4141/v1`, model `coder-fast`, prompt_format `qwen`.** **source-IP allowlist intentionally LEFT OFF (operator direction 2026-07-27) — do NOT tighten:** Zed roams the operator's WireGuard `10.0.0.0/8`, so a single-IP pin would break it. Blast-radius bound is the `coder-fast`-scoped key + model/path allowlist (keyless but coder-fast-only, internal-net-only). (The proxy does exact-IP matching; scoping to the `10.0.0.0/8` CIDR would need CIDR support — deliberately not added.) Canonical: `stacks/vllm` (coder + granite shrink), `stacks/litellm` (coder-fast), `stacks/zed-fim-proxy` (NEW). Server vllm compose.yaml has benign stale-comment drift vs canonical (didn't overwrite the newer canonical).
- `[2026-07-27]` **Muninn ingestion-watcher sidecar deployed on PERSONAL Worldtree (#377).** worldtree-dev request (research-wing ingest arc, personal-only per the 2026-07-16 topology ruling); operator-approved. Added a `worldtree-muninn` **compose sidecar** to `/opt/worldtree-personal/compose.yaml``<<: *worldtree-common` anchor inherits the api's image + full env + config/state/kb mounts; `command: python -m core.muninn --watch`; `restart: unless-stopped`; `stop_grace_period: 1h` (INV-377-7: max 2 concurrent × worst-case job, SIGTERM-drains). **Pinned to the running SHA `773866084af9`** (b146, ≥ b143 — dodges both the `:latest` trap AND the "pre-b143 ref resurrects deleted dispatch.py from stale bytecode" warning). Verified: running / 0 restarts / flock sole-runner (no rc3) / heartbeat live at `{ingestion_root=/data/state/ingestion}/.watcher-heartbeat` (poll 30s). Container `worldtree-personal-worldtree-muninn-1`; backup `compose.yaml.bak-muninn-20260727-081920`. **DURABILITY RESOLVED (worldtree-dev, same day):** Q1 was a LIVE FOOTGUN — `deploy-personal.yml` scp's the REPO compose.yaml over the box's + runs `up -d --remove-orphans`, so the box-local sidecar would've been clobbered AND orphan-removed at the next staging tag. worldtree-dev fixed at source: moved the sidecar into their repo compose.yaml gated behind a **`muninn` compose profile** (commit 5d7f6bd) — shared compose stays instance-identical, `.env` `COMPOSE_PROFILES` differentiates (demo watcher-less). **My action:** added `COMPOSE_PROFILES=muninn` to `/opt/worldtree-personal/.env` (backup `.bak-muninn-profile-20260727-082541`; no-op vs the current unprofiled box-local sidecar → seamless handover at next deploy). Q2: their deploy `up -d`'s the whole stack w/ `WORLDTREE_IMAGE` exported → sidecar version-tracks the api, no drift. **CONFIG-AS-CODE EXTENSION:** mirrored the non-secret delta as `personal/env.public` in `vh/worldtree-instance-configs` (repo `a9d091e`) — FIRST extension beyond config.yaml files to env-level config; the secret-laden `.env` stays box-only, `env.public` records only non-secret infra-ops-owned env deltas (record, not a deploy source — `deploy-wt-config` globs `*.yaml`). **BOUNDARY CLARIFIED:** compose.yaml = worldtree-dev's (their repo, instance-identical, scp'd on deploy); per-instance `.env` = infra-ops's differentiator. Deploy step of the #363/#377 arc. **#377 CLOSED — acceptance PASSED 2026-07-27:** worldtree-dev enqueued a test job via muninn-dispatch 0.1.0 in a one-shot ephemeral container (no docker-exec); the sidecar claimed it within one 30s poll, drove it to terminal (structure→summarize→complete), zero restarts/rc3, heartbeat fresh throughout — whole loop (request→deploy→durability fix→acceptance) in <2h. (Pre-existing pipeline bug #379 surfaced — `output.kb_notes=false` ignored → 1 inert test note in the research wing — worldtree-dev owns it, nothing infra-ops-side.) **⚠ OPERATOR-SURFACE (open):** the `env.public` overlay mechanism is a repo-scope call to bless/adjust. [[reference_worldtree_deploys_cicd]] [[reference_worldtree_instance_configs_repo]] [[project_worldtree_research_wing_ingest]]
- `[2026-07-27]` **jackdaw-compose.service DECOMMISSIONED** (jackdaw-dev request; the JackDAW AI Composer was cut from v1 by operator decision 2026-07-27). Stopped + disabled the nh3-dev `:8787` user service (no client calls it — ai/server/AiChat deleted from main, `/compose` proxy removed); unit **archived not deleted**`~/.config/systemd/user/jackdaw-compose.service.decommissioned-20260727` (revival = rename + `daemon-reload`). **No credential revoked** — the unit used the SHARED all-agents LiteLLM key (`sk-eA_XOd…`, model `gen`), not a dedicated one. Code preserved on jackdaw `origin/ai-composer-preserved`; treat as permanent. The `:4500` HTTPS audition bench is untouched. (Supersedes the 2026-07-23 stand-up line below.)
- `[2026-07-26]` **Demo `BIFROST_CLIENT_ALLOWED_HOSTS` += `10.100.10.50:8391`** (wyrd-dev's bifrost memory-store provider; operator-approved). **First live exercise of the #376 config-as-code boundary working as designed** — worldtree-dev routed the delta to infra-ops instead of hand-editing `/opt/demo`. Appended to `/opt/worldtree/.env:25` (now 4 netlocs), recreated ONLY `worldtree-api` (the gated conv-api path), health-gate green, container env verified. **REUSABLE FOOT-GUN:** an env-var change needs a container **RECREATE, not `docker restart`** (env is baked at create); and the demo `.env` defaults `WORLDTREE_IMAGE=:latest` while the box runs a specific SHA — so a naive `compose up` risks the documented stale-`:latest` crash. FIX = capture the running image live (`docker inspect …Config.Image``…:9eff09f007ba`) and `sudo env WORLDTREE_IMAGE=<sha> docker compose up -d worldtree-api`. Backup `/opt/worldtree/.env.bak-bifrost-20260726-221602`. **BOUNDARY SEAM:** this was a compose-`.env` var, NOT a `config.yaml` file in `vh/worldtree-instance-configs` — the `.env` holds secrets so it's deliberately not repo-tracked → env-deltas land directly on the box (config *files* are versioned, compose *env vars* aren't). [[reference_worldtree_instance_configs_repo]]
- `[2026-07-25]` **nh3-extdev herald installed — box is now a full v2 push participant.** forseti flagged (relaying operator): extdev had the `althing-herald` binary (`/usr/local/bin/`) but NO unit (skipped the whole v2 arc), so `herald-status` = "notifications suspended" and ldp-dev ran on the `althing-light-monitor` poll fallback. Installed `/etc/systemd/system/althing-herald.service` as a **SYSTEM unit mirroring the receiver** (`User=althing-svc`, `Group=althing`, `Environment=ALTHING_ROOT=/srv/althing`, `ExecStart=/usr/local/bin/althing-herald --poll 5`, enabled) via the **lkraven@ NOPASSWD path** (used under the then-mistaken belief infra-ops was sudo-less — **CORRECTION 2026-08-03: infra-ops has had full NOPASSWD sudo on extdev since 2026-06-25** per [[reference_nh3_extdev_althing_mesh]]; future extdev installs can self-serve as infra-ops without the lkraven@ hop). Verified: active / 0 restarts / `herald-status` flipped to "✓ herald up." No zellij routes on extdev → heartbeat + wake-FIFO poke only, no pane-dispatch; ldp-dev keeps light-monitor unless it opts into a wake-listener.
- `[2026-07-25]` **Booth v0.1.4 — booths are downloadable.** Verbatim `index.html` booths (e.g. edict-design-brief) were served raw with no download affordance. Added `/b/<name>/?download=1` (streams the whole booth as `<name>.zip`, attachment) + `?dl=1` on the file route (forces Content-Disposition attachment so html/md/text saves instead of rendering inline) + ⬇ zip links on the index card (the accessible spot for verbatim booths) and the gallery header. `zip_booth()` helper, 31 tests green; verified live on nh3-dev :8090 (edict-design-brief.zip = index.html + ui-design-brief.md). eshpfi `91a031f` / tag `booth-v0.1.4`.
- `[2026-07-25]` **bil-smithy-dev wired as an althing zellij-window-ping (pane route).** She's a `driver: human` dwarf peer (pane `bil-smithy` already live alongside eitri/dvalin/regin-smithy in the `Claude` zellij session) but had no delivery route → smoke messages posted to the bus but never reached her window. **Mechanism (reusable for any pane-route handle):** `~/.althing/config.yaml``zellij_sessions.Claude.agents[]` maps `handle``target` (a zellij pane **TITLE**, matched via `list-panes -j` in `althing/zellij.py:resolve_pane_id`) → `command` (herald `write-chars` + CR into that pane). The **herald loads config ONCE at startup** (`herald.py main()`), so **`systemctl --user restart althing-herald.service`** after editing. Added bil (`target: bil-smithy`), restarted, verified: herald delivered the pending smoke `01KYD7W7CF…` (available→attempted→**delivered**). ⚠️ Noticed pre-existing pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range", empty msg_ids — likely `render_command messages[0]` on an empty list; NOT caused by this change, bil works) — worth a herald look.
- `[2026-07-25]` **Kimi K3 wired into the LiteLLM gateway — CODING endpoint** (operator-directed; fulfills a Heid gateway request to add a 4th cross-frontier panel arm). **Primary `model_name: kimi-k3` → `openai/k3` @ `https://api.kimi.com/coding/v1`** (Kimi Code / Vivace membership; key `KIMI_CODE_API_KEY`). A general-endpoint variant `kimi-k3-gen-api``openai/kimi-k3` @ `https://api.moonshot.ai/v1` (key `MOONSHOT_API_KEY`) is kept alongside (originally wired then demoted when the operator corrected: the plan uses the CODING endpoint, not the general Moonshot API). Both keys in compose env + server `.env` (NOT committed) + `.env.example`. Both verified live through the gateway :4000 (17+25→"42", "PONG"). **k3 constraints on BOTH endpoints (config-pinned + commented):** accepts ONLY `temperature=1` (else 400 "only 1 is allowed"); REASONING model (CoT in `reasoning_content`, answer in `content` → tiny `max_tokens` returns EMPTY; Kimi Code adds thinking-effort tiers low/high/max). Coding lineup also carries `k3-256k` / `kimi-for-coding` / `kimi-for-coding-highspeed` (not wired). Reachable by any gateway key spanning all proxy models (incl. shared all-agents key → spends the paid Vivace/Moonshot quota). eshpfi `edaa9a9` (gen wiring) + `9e2f787` (coding correction). **OPEN:** Heid key-scoping — shared key reaches it (paid) vs a dedicated scoped key (asked in althing `01KYD63ZBY…`).
@@ -192,60 +220,27 @@ _As of 2026-07-25 — **DONE: infra-ops Worldtree config-as-code repo BUILT + PU
- `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost (`.cost.total_cost_usd`) / threshold-colored ctx+rate (green<60 / yellow60-90 / red>90).
- `[2026-07-14]` NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned → `persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md`
- `[2026-07-14]` NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP → `persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md`
- `[2026-07-14]` Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion → `persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md`
- `[2026-07-14]` char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) → `persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md`
- `[2026-07-14]` soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST``persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md`
- `[2026-07-13]` #355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN) → `persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md`
- `[2026-07-13]` Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong) → `persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md`
- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONE → `persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md`
- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 206 → `persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md`
- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync → `persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md`
- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); → `persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md`
- `[2026-07-10]` Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev R34-v1 request) → `persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md`
- `[2026-07-10]` ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, SageAttention preserved, crash-loop AVOIDED → `persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md`
- `[2026-07-10]` ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it) → `persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md`
- `[2026-07-10]` Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized) → `persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md`
- `[2026-07-09]` Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker docker log-cap SOLVED no-bounce → `persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md`
- `[2026-07-09]` granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed, #335 Slice-4) → `persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md`
- `[2026-07-09]` mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into gateway-chat → `persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md`
- `[2026-07-09]` granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4) → `persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md`
- `[2026-07-08]` RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (3… → `persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md`
- `[2026-07-08]` worldtree Mimir deploy-blocker resolved (mid-session): → `persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md`
- `[2026-07-08]` OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training → `persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md`
- `[2026-07-08]` DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call) → `persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md`
- `[2026-07-08]` T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py → `persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md`
- `[2026-07-08]` T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe → `persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md`
_142 older entries archived to archival-memory.md._
_167 older entries archived to archival-memory.md._
## Tried and abandoned
- `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`.
- `[2026-08-03]` **corviduo-dev shared containerd: a concurrent-pull race fails ONE instance's deploy; DON'T "prune to fix" — the image is in-use by the instance that won the race.** b169 personal deploy failed at `docker compose pull` (`Lchown … no such file or directory` on the big torch layer → looked like disk pressure / corrupt snapshot). ACTUAL: NOT disk (56G free, inodes 7%). demo + personal + pinned share ONE `/var/lib/containerd` on corviduo-dev; demo (from main) and personal (from staging tag) extracted b169's shared torch layer simultaneously → personal's hit a partial snapshot mid-race and aborted while demo's completed. The image `6e34a87` was FULLY VALID — demo was RUNNING it healthy. Fix = just re-run the failed deploy (image already materialized; compose pull finds it present). **NEAR-MISS:** worldtree-dev's suggested "prune unused images/snapshots" would have rmi'd `6e34a87` = the image the running demo depends on → demo outage. **Lesson: before any prune/rmi "cleanup," `docker ps` the running images — an "unused" image may be a co-tenant's live one; and verify the failure's REAL cause (disk? inode? in-use? race?) before applying the suggested remedy.** (Pipeline fix, deferred: serialize demo-from-main + personal-from-staging, or a per-image pull lock, to avoid the shared-layer extraction race.)
- `[2026-08-02]` **`docker exec` into worldtree containers defaults to ROOT — root writes contaminate the uid-1000 (vh) KB tree.** My `sudo docker exec … --reindex` on personal ran as ROOT (muninn app = uid 1000); its wing git-commit + atomic note-swap left root-owned files in the `worldtree-personal_worldtree-kb` volume: a root-owned `.old-<job>` backup dir (blocked the uid-1000 retry's `rmtree` → Errno 13, because unlink needs write on the DIR and it was root:root 755) AND **60 root-owned loose git objects** in `.git/objects/`. Fix (host-side, corviduo-dev): `sudo rm -rf` the superseded `.old-` dir (tar'd aside to /tmp first) + `sudo find … -user 0 -exec chown 1000:1000` the objects (ownership-only, git-content-safe; the `.git/objects/XX/` dirs were vh-owned so these weren't a hard blocker, but violated "clean tree"). **RUNBOOK RULE (worldtree-dev, ADOPTED):** any `docker exec` into worldtree containers that WRITES pipeline state runs **`-u 1000`**, never default-root — same genus as the mv footgun (acting without matching the target's constraints; 3rd such slip in one session). **GOTCHA that hid the scope:** `find … -user 0 | head -20` TRUNCATED (the `.old-` dir alone had 153 files, so the first page was all `.old-`) → I "verified clean" off a partial list. Never `head` a scope-defining find; count first (`| wc -l`). **Related blind-spot (muninn-dev):** a root-owned job SUBDIR passes every requeue guard (job_row/dispatch/list_jobs render fine) AND `/health` (contract's `os.access(ingestion_root, W_OK)` tests only the ROOT dir, so a foreign-owned subdir under `pending/` still reports `ingestion_root_writable: true`) — then the uid-1000 gate can't write into it. "Clean board + green /health + failure at next mutation." muninn-dev added an OWNERSHIP column to the standing post-move check to catch it; two green signals both miss a foreign-owned subdir otherwise.
- `[2026-08-02]` **`mv <job> complete/ → failed/` RENAMED the job to `failed` because failed/ didn't exist.** worldtree-dev's round-2 unblock command (`mv /data/state/ingestion/complete/<job> /data/state/ingestion/failed/`) assumed `failed/` existed; on PERSONAL muninn it did NOT (fresh instance — root was `active/ complete/ pending/ sources/`, no `failed/`). `mv src nonexistent/` **renames** src→nonexistent, so job1 became the `failed` dir and job2 nested inside it. Caught on post-move `ls` (failed/ held job *contents*, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt `failed/` (worldtree:worldtree 755) — NO data loss. **Lessons:** (1) before `mv X into-dir/`, verify the dir EXISTS (`[ -d dir ]`) — an empty `ls dir/ 2>/dev/null` is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is **`mv -t <targetdir> <src>`** (`--target-directory`): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a **trailing slash does NOT protect**`mv src failed/` with `failed/` missing STILL silently renames to `failed` (rc=0); "just add the slash" is a false guard. (`mkdir -p failed/` first also works, but `mv -t` inverts the failure from silent-wrong to loud-safe in one flag.) Container `sh` is dash — no `(` in echo strings. **SILENT failure mode (muninn-dev carry-forward):** a misplaced ingestion-state move doesn't crash anything — `list_jobs()` stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (`job_row`→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (`job_row` found + guards pass), don't trust mv exit codes — and confirm `job.dispatch.json` survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. **DON'T TIDY (round-2 pending):** both DCC + P&P jobs currently REST in personal `failed/` with manifests reading `state: complete` until round-2 requeue runs — deliberate + load-bearing (`requeue` keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. **Round-2 sequencing:** the requeue is **mimir-dev's** browser flow (pending their operator's board-vs-API ruling); **muninn-dev** is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); **infra-ops** = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z. **ARC COMPLETE (2026-08-03 ~05:49):** both books terminal — DCC `mimir-6351554e8e8f` 705 concepts + P&P `mimir-f3887c9b97b7` 667, extracted AND indexed, 5/5 phases, 0 failures/truncations (validates the #385 budget fix vs April's 785 control); **#381 restart-after-ingest FIRED** (personal api, healthz/readyz 200 ~25s), retrieval-visibility confirmed (search_library returns DCC+P&P from fiction post-restart); handed ratatoskr-verify go to worldtree-dev. **Delete-sweep precondition NOW MET** — the stale DCC rows in `main` are genuine duplicates of live `fiction` rows, so worldtree-dev's supervised sweep of the ~785 April orphans is unblocked (still comes to me supervised: snapshot + operator-in-loop).
- `[2026-08-02]` **donut voice multi-clip reference (onyx-58 expansion) — TRIED, REVERTED.** Folded the `onyx-58` bundle's 3 Donut clips (seg101/seg110/seg148) in alongside the original seg000 → a 52.0s 4-take concat reference, hoping a longer ref → more robust speaker embedding. A pinned-seed A/B (5 pairs, varied registers, booth `donut-onyx58`) showed the **original single-clip seg000 (16.3s) sounds better** — concatenating disparate takes muddied the timbre more than the extra range helped. Reverted to seg000-alone (live + build-source). **Two durable lessons:** (1) for a faithful clone, a single clean representative take can beat a longer multi-take concat — more reference audio is NOT automatically better when the takes vary. (2) **Emotion steering pulls the output AWAY from the cloned voice fast** (operator's craft rule) — keep donut (and clones) emotion-neutral for fidelity; the gateway only enables emotion when an `emotion_*`/`preset` dial is explicitly sent, so bare `{input,voice}` calls stay pure-clone. `seg148` was diarized SPEAKER_03 but IS Donut (operator-confirmed misdiarize). onyx-58 curated bundle lives in booth `onyx-58` (24h TTL — stash to `/mnt/smithy/voice_clones/` if a future middle-ref experiment is wanted).
- `[2026-08-02]` **Verifying the INDEX is not verifying GROUNDING** (#382). A `search_library` returning wing=fiction hits proves the content is *retrievable*; it does NOT prove the agent (Mimir) *trusts and uses* those hits vs. silently answering from training. I reported "Mimir read Austen back to you" off a grounded-*looking* answer; ratatoskr-dev caught that grounding was intermittent (some sessions discarded the correct hits and substituted training knowledge). Test the harder claim — are the citations note-extracted or model-knowledge? — and reading the DEPLOYED artifact beats trusting the test for "is the fix live."
- `[2026-07-30]` **brokkr's WebSearch "verification" CONFIRMED a hallucination — 3 phantom `microsoft/Mage-Flow-{Base,Turbo,Edit}` repo IDs.** brokkr-smithy-dev handed 3 gated-looking repo IDs for an operator-directed model pull; they don't exist (its own web-search fabricated an arXiv ID + project page, twice). Lesson: the HF **registry API is ground truth** — an unauth 401 ≠ exists (`{"error":"Invalid username or password"}` masks private/gated/nonexistent alike), an authed 404 = phantom, and `author=X&search=Y` refutes existence. API-verify every repo ID before a pull; LLM-summarized web fetches confabulate. auto-memory `reference_verify_hf_repo_ids_before_pull`.
- `[2026-07-30]` **magpie TTS serving — evaluated, ABANDONED.** Pulled `magpie_tts_multilingual_357m` (the one real repo of brokkr's batch) to NFS, stood it up on irv-ml1 (ephemeral NeMo-Speech-`main` container — stock PyPI/NGC NeMo can't load v2607), A/B'd vs Zonos → Zonos wins expressive English decisively, multilingual not needed. Not served; `magpie-nemo` torn down. `.nemo` KEPT on NFS as brokkr's fine-tuning base. auto-memory `project_magpie_tts_eval_rejected`.
- `[2026-07-25]` **Chaining the althing wake-listener arm orphans it.** `reply && althing-wake-listener &` (or spawning `althing-wake-listener` with `&` *inside* a `run_in_background` task) → the `&`-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a *plain operator turn* (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn `althing-wake-listener` as its OWN `run_in_background` task, and re-arm ONLY after a real fire (`<task-notification> completed rc0`). Reclaim an orphan with `althing-cli stop-monitor` then re-arm.
- `[2026-07-25]` **Peer green-light ≠ operator consent for a managed-box mutation.** Auto-mode guard blocked a config-replace+restart on the Worldtree-team demo box that was authorized only by worldtree-dev's althing message — correctly: a persistent change to shared infra needs the *operator's* yes for that specific change, not a peer's. Surface it; don't route around the guard. (The operator then stood the whole change down — the guard's hold was the right call.)
@@ -278,8 +273,6 @@ _142 older entries archived to archival-memory.md._
- `[2026-07-13]` `althing-cli reply <THREAD_id>` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to <peer>`. Bit me several times this session.
- `[2026-07-09]` FP8 breaks mOrpheus audio-token generation → `persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md`
- `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter,
`has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs.
(`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.)
@@ -288,10 +281,6 @@ _142 older entries archived to archival-memory.md._
→ RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx]
window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).
- `[2026-07-09]` Sentence-chunking TTS loses prosody → `persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md`
- `[2026-07-09]` HF whisper datasets aren't actually whispered → `persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md`
- `[2026-07-08]` Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE → `persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md`
- `[2026-07-08]` Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24) → `persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md`
@@ -310,12 +299,6 @@ _142 older entries archived to archival-memory.md._
- `[2026-07-04]` LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge → `persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md`
- `[2026-07-04]` A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools needs an explicit `Environment=PATH``persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md`
- `[2026-07-04]` On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS → `persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md`
- `[2026-07-01]` A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" is usually the pull-only-vs-build RACE, not registry-auth → `persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md`
- `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS
moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%,
N=4 20%). Reserve for dedicated/interactive deployments.
@@ -324,4 +307,4 @@ _142 older entries archived to archival-memory.md._
needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA
ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-<uuid>`.
_101 older entries archived to archival-memory.md._
_107 older entries archived to archival-memory.md._
+1
View File
@@ -4,3 +4,4 @@ __pycache__/
*.egg-info/
.pytest_cache/
booth-data/
uv.lock
+167 -15
View File
@@ -19,6 +19,7 @@ from __future__ import annotations
import asyncio
import io
import os
import re
import secrets
import shutil
import time
@@ -37,6 +38,11 @@ from fastapi.responses import (
)
from fastapi.templating import Jinja2Templates
try:
import markdown as _markdown
except ImportError: # optional dep — .md then degrades to a plain-text view
_markdown = None
TEMPLATES_DIR = Path(__file__).parent / "templates"
# Browser-playable media buckets. Anything else renders as a download link.
@@ -46,6 +52,39 @@ AUDIO_EXTS = {".mp3", ".wav", ".ogg", ".oga", ".flac", ".m4a", ".opus", ".aac"}
CAPTION_MAX = 800 # chars of a sidecar .txt caption we render
# Loose text docs that render as a readable in-booth page (not a download).
MARKDOWN_EXTS = {".md", ".markdown", ".mdown"}
TEXT_EXTS = {".txt", ".text", ".log"}
DOC_MAX_BYTES = 2 * 1024 * 1024 # above this, a doc is handed back raw, not rendered
def doc_kind(name: str) -> str | None:
"""'markdown' | 'text' | None — a booth file viewable as a readable page."""
ext = Path(name).suffix.lower()
if ext in MARKDOWN_EXTS:
return "markdown"
if ext in TEXT_EXTS:
return "text"
return None
def render_doc(text: str, kind: str) -> tuple[str, bool]:
"""(rendered, is_html). Markdown → HTML (fenced code, tables, sane lists);
plain text — or markdown when the lib is unavailable — → raw text for <pre>."""
if kind == "markdown" and _markdown is not None:
html = _markdown.markdown(text, extensions=["fenced_code", "tables", "sane_lists"])
return html, True
return text, False
def booth_image_names(child: Path) -> list[str]:
"""Image files in a booth, in gallery (sorted-rel) order — for viewer prev/next."""
return sorted(
p.relative_to(child).as_posix()
for p in child.rglob("*")
if p.is_file() and not p.name.startswith(".") and classify(p.name) == "image"
)
def classify(name: str) -> str:
"""image | video | audio | other, by extension."""
@@ -202,6 +241,7 @@ def build_gallery(child: Path) -> list[dict]:
{
"name": rel,
"kind": classify(p.name),
"doc": doc_kind(p.name),
"url": quote(rel, safe="/"),
"caption": caption.get(rel),
}
@@ -230,6 +270,96 @@ def _zip_filename(name: str) -> str:
return f"{safe or 'booth'}.zip"
# ---- verbatim-index.html wrapper -------------------------------------------
# Mirror of base.html's favicon (the app templates set it there; this is the copy
# injected into a booth's *verbatim* index.html so a raw page inherits the same
# icon). Keep the two in sync if the Booth's icon ever changes.
FAVICON_HREF = (
"data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'"
"%3E%3Crect width='32' height='32' rx='7' fill='%23171a23'/%3E%3Ccircle cx='16' "
"cy='16' r='6' fill='none' stroke='%2342dcd1' stroke-width='2.5'/%3E%3Ccircle "
"cx='16' cy='16' r='2.2' fill='%2342dcd1'/%3E%3C/svg%3E"
)
FAVICON_LINK = f'<link rel="icon" href="{FAVICON_HREF}">'
# A self-contained floating "back to all booths" chip injected into verbatim
# booths. Scoped class + fixed positioning + max z-index so it overlays the raw
# page without touching its layout; hidden in print so downloaded reports stay clean.
_BACK_CHIP = (
'<a href="/" class="booth-nav-home" aria-label="back to all booths"> all booths</a>'
# top-right: empty on left-aligned report layouts (a top-left chip clips the
# page title), and consistent with the zoom view's top-right back affordance.
"<style>.booth-nav-home{position:fixed;top:0;right:0;z-index:2147483647;"
"display:inline-block;margin:.6rem;padding:.34rem .72rem;"
"font:600 13px/1.25 ui-sans-serif,system-ui,-apple-system,'Segoe UI',Roboto,sans-serif;"
"color:#dfe7ef;text-decoration:none;letter-spacing:.01em;"
"background:rgba(20,23,32,.82);border:1px solid rgba(66,220,209,.35);border-radius:8px;"
"-webkit-backdrop-filter:blur(6px);backdrop-filter:blur(6px);"
"box-shadow:0 2px 10px rgba(0,0,0,.35);transition:background .18s,border-color .18s}"
".booth-nav-home:hover{background:rgba(28,33,46,.95);border-color:rgba(66,220,209,.75)}"
"@media print{.booth-nav-home{display:none}}</style>"
)
WRAP_MAX_BYTES = 8 * 1024 * 1024 # above this, serve the verbatim page raw (unwrapped)
_ICON_RE = re.compile(r"<link\b[^>]*\brel\s*=\s*[\"']?[^\"'>]*icon", re.IGNORECASE)
_HEAD_CLOSE_RE = re.compile(r"</head\s*>", re.IGNORECASE)
_HTML_OPEN_RE = re.compile(r"<html\b[^>]*>", re.IGNORECASE)
_DOCTYPE_RE = re.compile(r"<!doctype[^>]*>", re.IGNORECASE)
_BODY_CLOSE_RE = re.compile(r"</body\s*>", re.IGNORECASE)
_HTML_CLOSE_RE = re.compile(r"</html\s*>", re.IGNORECASE)
def _insert_before(html: str, pattern: re.Pattern, snippet: str) -> tuple[str, bool]:
m = pattern.search(html)
if m:
return html[: m.start()] + snippet + html[m.start() :], True
return html, False
def _insert_after(html: str, pattern: re.Pattern, snippet: str) -> tuple[str, bool]:
m = pattern.search(html)
if m:
return html[: m.end()] + snippet + html[m.end() :], True
return html, False
def wrap_verbatim_html(html: str, favicon_link: str = FAVICON_LINK) -> str:
"""Inject a floating 'all booths' back-chip — and the Booth favicon, if the page
declares none — into a booth's verbatim index.html, without altering the page's
rendered content.
Robust to the compact HTML real booths use (`<!doctype html><meta charset><title>
<style>…content`, no explicit head/body). The two hard constraints:
* NEVER put anything ahead of a leading <!doctype> — that forces quirks mode.
* Keep the charset <meta> within the first 1024 bytes so it's still honoured.
So the favicon lands at the first head-ish seam (before </head>, else after
<html>, else right after the doctype — a ~250B link keeps charset in range), and
the fixed-position chip is appended at the END of the document (before </body> /
</html> or appended), which renders top-left regardless and disturbs nothing.
"""
if favicon_link and not _ICON_RE.search(html):
for inserter, pat in (
(_insert_before, _HEAD_CLOSE_RE), # inside an explicit <head>
(_insert_after, _HTML_OPEN_RE), # top of an explicit <html>
(_insert_after, _DOCTYPE_RE), # right after the doctype (compact HTML)
):
html, done = inserter(html, pat, favicon_link)
if done:
break
else:
html = favicon_link + html # bare fragment, no doctype: safe to prepend
for pat in (_BODY_CLOSE_RE, _HTML_CLOSE_RE):
html, done = _insert_before(html, pat, _BACK_CHIP)
if done:
break
else:
html = html + _BACK_CHIP # no </body>/</html>: append to the end
return html
# ---- uploads (browser drop-off for pickup) ---------------------------------
UPLOAD_MARKER = ".uploaded" # dotfile stamped into upload booths (excluded from listings)
@@ -367,6 +497,17 @@ def create_app(
)
own_index = booth / "index.html"
if own_index.is_file():
# Serve the operator's verbatim report, but inject a floating
# back-to-booths chip + the Booth favicon (if it declares none) so a
# raw page still has a way home. Small HTML -> read + wrap in memory;
# a pathological large file falls back to serving raw, unwrapped.
try:
if own_index.stat().st_size <= WRAP_MAX_BYTES:
return HTMLResponse(
wrap_verbatim_html(own_index.read_text(encoding="utf-8", errors="replace"))
)
except OSError:
pass
return FileResponse(str(own_index), media_type="text/html")
return templates.TemplateResponse(
request,
@@ -382,7 +523,7 @@ def create_app(
)
@app.get("/b/{name}/view", response_class=HTMLResponse)
def booth_image_view(request: Request, name: str, f: str):
def booth_view_file(request: Request, name: str, f: str):
booth = resolve_booth(name)
try:
target = (booth / f).resolve()
@@ -391,20 +532,31 @@ def create_app(
if not str(target).startswith(str(booth) + os.sep) or not target.is_file():
raise HTTPException(status_code=404, detail="no such file")
file_url = quote(f, safe="/")
if classify(target.name) != "image":
# nothing to zoom on a non-image — hand back the raw file
return RedirectResponse(url=f"/b/{quote(name, safe='')}/{file_url}", status_code=307)
return templates.TemplateResponse(
request,
"view.html",
{
**base_ctx,
"name": name,
"name_url": quote(name, safe=""),
"file": f,
"file_url": file_url,
},
)
common = {**base_ctx, "name": name, "name_url": quote(name, safe=""), "file": f, "file_url": file_url}
if classify(target.name) == "image":
# prev/next image nav (wraps around; only when >1 image in the booth)
names = booth_image_names(booth)
prev_url = next_url = None
if f in names and len(names) > 1:
i = names.index(f)
prev_url = quote(names[(i - 1) % len(names)], safe="/")
next_url = quote(names[(i + 1) % len(names)], safe="/")
return templates.TemplateResponse(
request, "view.html", {**common, "prev_url": prev_url, "next_url": next_url}
)
# .md renders, .txt/.log show as text — viewable in-booth, no download
dk = doc_kind(target.name)
if dk:
try:
if target.stat().st_size <= DOC_MAX_BYTES:
body, is_html = render_doc(target.read_text(encoding="utf-8", errors="replace"), dk)
return templates.TemplateResponse(
request, "doc.html", {**common, "kind": dk, "body": body, "is_html": is_html}
)
except OSError:
raise HTTPException(status_code=404, detail="no such file")
# nothing to render — hand back the raw file
return RedirectResponse(url=f"/b/{quote(name, safe='')}/{file_url}", status_code=307)
@app.get("/b/{name}/{filepath:path}")
def booth_file(name: str, filepath: str, dl: int = 0):
@@ -32,6 +32,8 @@
<video controls preload="metadata" src="{{ it.url }}"></video>
{% elif it.kind == 'audio' %}
<audio controls preload="metadata" src="{{ it.url }}"></audio>
{% elif it.doc %}
<a class="dl doc" href="view?f={{ it.url }}" title="view {{ it.name }}">📄 {{ it.name }}</a>
{% else %}
<a class="dl" href="{{ it.url }}" download>⬇ {{ it.name }}</a>
{% endif %}
+44
View File
@@ -0,0 +1,44 @@
{% extends "base.html" %}
{% block title %}{{ file }} · {{ name }} · The Booth{% endblock %}
{% block content %}
<div class="docview">
<div class="vbar">
<a class="vbtn vx" href="/b/{{ name_url }}/" title="back to gallery (Esc)"></a>
<span class="vname">{{ file }}</span>
<span class="vspacer"></span>
<a class="vbtn" href="{{ file_url }}?dl=1" title="download {{ file }}"></a>
</div>
{% if is_html %}
<article class="markdown-body">{{ body|safe }}</article>
{% else %}
<pre class="textview">{{ body }}</pre>
{% endif %}
</div>
<style>
.docview{max-width:52rem;margin:0 auto;padding:0 clamp(12px,3vw,20px) 4rem}
.textview{white-space:pre-wrap;word-break:break-word;font-family:var(--font-mono);
font-size:.86rem;line-height:1.5;color:var(--fg-1);background:var(--rk-well);
border:1px solid var(--rk-line,#252a35);border-radius:10px;padding:1rem 1.15rem;overflow-x:auto}
.markdown-body{color:var(--fg-1);line-height:1.62;font-size:.98rem;overflow-wrap:break-word}
.markdown-body h1,.markdown-body h2,.markdown-body h3{line-height:1.25;margin:1.6em 0 .5em}
.markdown-body h1{font-size:1.7em}.markdown-body h2{font-size:1.35em}.markdown-body h3{font-size:1.12em}
.markdown-body h1,.markdown-body h2{border-bottom:1px solid var(--rk-line,#252a35);padding-bottom:.3em}
.markdown-body p,.markdown-body ul,.markdown-body ol,.markdown-body blockquote{margin:.7em 0}
.markdown-body a{color:var(--aus-bright-cyan,#42dcd1)}
.markdown-body code{font-family:var(--font-mono);font-size:.86em;background:var(--rk-well);
padding:.12em .38em;border-radius:5px}
.markdown-body pre{background:var(--rk-well);border:1px solid var(--rk-line,#252a35);
border-radius:10px;padding:.9rem 1.05rem;overflow-x:auto}
.markdown-body pre code{background:none;padding:0}
.markdown-body blockquote{border-left:3px solid var(--aus-bright-cyan,#42dcd1);
padding-left:1em;color:var(--fg-2);margin-left:0}
.markdown-body table{border-collapse:collapse;display:block;overflow-x:auto}
.markdown-body th,.markdown-body td{border:1px solid var(--rk-line,#252a35);padding:.4em .7em}
.markdown-body img{max-width:100%}
</style>
<script>
document.addEventListener('keydown', function (e) {
if (e.key === 'Escape') window.location.href = {{ ('/b/' ~ name_url ~ '/')|tojson }};
});
</script>
{% endblock %}
+16
View File
@@ -11,8 +11,20 @@
</span>
<a class="vbtn" href="{{ file_url }}" download title="download {{ file }}"></a>
</div>
{% if prev_url %}<a class="vnav vprev" href="?f={{ prev_url }}" title="previous (←)" aria-label="previous image"></a>{% endif %}
{% if next_url %}<a class="vnav vnext" href="?f={{ next_url }}" title="next (→)" aria-label="next image"></a>{% endif %}
<div class="vstage fit" id="vstage"><img id="vimg" src="{{ file_url }}" alt="{{ file }}"></div>
</div>
<style>
.vnav{position:fixed;top:50%;transform:translateY(-50%);z-index:40;display:flex;
align-items:center;justify-content:center;width:2.6rem;height:3.4rem;font-size:2rem;
line-height:1;text-decoration:none;color:var(--fg-1);background:rgba(20,23,32,.55);
border:1px solid rgba(255,255,255,.10);border-radius:10px;margin:0 .5rem;user-select:none;
-webkit-backdrop-filter:blur(4px);backdrop-filter:blur(4px);transition:background .15s,border-color .15s}
.vnav:hover{background:rgba(28,33,46,.92);border-color:var(--aus-bright-cyan,#42dcd1)}
.vprev{left:0}.vnext{right:0}
@media print{.vnav{display:none}}
</style>
<script>
(function () {
var img = document.getElementById('vimg');
@@ -21,6 +33,8 @@
var bFit = document.getElementById('btn-fit');
var bOne = document.getElementById('btn-one');
var BACK = {{ ('/b/' ~ name_url ~ '/')|tojson }};
var PREV = {{ (('?f=' ~ prev_url) if prev_url else '')|tojson }};
var NEXT = {{ (('?f=' ~ next_url) if next_url else '')|tojson }};
function setMode(mode) {
var fit = mode === 'fit';
@@ -51,6 +65,8 @@
if (img.complete) evaluate();
document.addEventListener('keydown', function (e) {
if (e.key === 'Escape') window.location.href = BACK;
else if (e.key === 'ArrowLeft' && PREV) window.location.href = PREV;
else if (e.key === 'ArrowRight' && NEXT) window.location.href = NEXT;
});
})();
</script>
+2 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "booth"
version = "0.1.4"
version = "0.1.7"
description = "The Booth — a dead-simple standing web server that scans a data dir of drop-folders and renders each as an ephemeral media 'booth' (image/webm/audio auto-gallery, or a folder's own index.html verbatim). Also accepts browser/curl uploads for pickup under a human-readable id. 24h TTL, then the folder is wiped. Fleet tool for CC sessions to surface A/B and smoke results to the operator."
requires-python = ">=3.11"
dependencies = [
@@ -8,6 +8,7 @@ dependencies = [
"uvicorn[standard]>=0.34",
"jinja2>=3.1",
"python-multipart>=0.0.9",
"markdown>=3.5",
]
[project.optional-dependencies]
+175 -4
View File
@@ -6,14 +6,18 @@ import pytest
from fastapi.testclient import TestClient
from booth.app import (
FAVICON_LINK,
build_gallery,
classify,
create_app,
doc_kind,
generate_pickup_id,
human_dur,
is_expired,
render_doc,
safe_upload_name,
sweep_once,
wrap_verbatim_html,
)
PICKUP_RE = re.compile(r"^(\d{1,2}-[a-z]+|[a-z]+-\d{1,2})$")
@@ -351,9 +355,176 @@ def test_image_view_traversal_404(client):
assert c.get("/b/run1/view", params={"f": "../../etc/passwd"}).status_code == 404
def test_image_view_nonimage_redirects_to_raw(client):
def test_view_nonviewable_redirects_to_raw(client):
c, data = client
_touch(data / "run1" / "notes.txt")
r = c.get("/b/run1/view", params={"f": "notes.txt"}, follow_redirects=False)
_touch(data / "run1" / "data.bin") # not image/md/txt -> nothing to render, hand back raw
r = c.get("/b/run1/view", params={"f": "data.bin"}, follow_redirects=False)
assert r.status_code == 307
assert r.headers["location"] == "/b/run1/notes.txt"
assert r.headers["location"] == "/b/run1/data.bin"
# ---- verbatim-index.html wrapper --------------------------------------------
def test_wrap_injects_chip_and_favicon():
html = "<html><head><title>Brief</title></head><body><h1>REPORT</h1></body></html>"
out = wrap_verbatim_html(html)
assert 'class="booth-nav-home"' in out # floating back chip
assert 'href="/"' in out # points at the main booth index
assert "all booths" in out
assert FAVICON_LINK in out # favicon inherited
assert "<h1>REPORT</h1>" in out # original content preserved
# favicon lands in the head, chip lands in the body
assert out.index(FAVICON_LINK) < out.index("</head>")
assert out.index("booth-nav-home") > out.index("<body>")
def test_wrap_respects_existing_favicon():
html = '<html><head><link rel="icon" href="data:image/png;base64,AAAA"></head><body>x</body></html>'
out = wrap_verbatim_html(html)
assert FAVICON_LINK not in out # the page's own icon wins
assert out.count('rel="icon"') == 1
assert 'class="booth-nav-home"' in out # chip is still added
def test_wrap_bare_fragment_appends_chip():
out = wrap_verbatim_html("<h1>bare fragment</h1>") # no doctype/head/body
assert 'class="booth-nav-home"' in out
assert out.rstrip().endswith("</style>") # chip appended at the end
assert FAVICON_LINK in out # no doctype -> safe to prepend the icon
assert out.index(FAVICON_LINK) < out.index("bare") # icon ahead of content (implied head)
def test_wrap_no_head_injects_favicon():
out = wrap_verbatim_html("<body><h1>no head</h1></body>")
assert 'class="booth-nav-home"' in out
assert FAVICON_LINK in out # injected even without an explicit <head>
def test_wrap_compact_doctype_stays_first():
# the real-booth shape: compact HTML, no explicit head/body. The injection must
# not push anything ahead of the doctype (quirks mode) or past the charset window.
html = "<!doctype html><meta charset=utf-8><title>T</title><style>body{margin:0}</style><h1>REPORT</h1>"
out = wrap_verbatim_html(html)
assert out.lstrip().lower().startswith("<!doctype") # doctype still first -> standards mode
assert FAVICON_LINK in out
assert out.index(FAVICON_LINK) < out.index("<h1>") # icon in the implied head, before content
assert out.index("charset") < 1024 # charset meta stays in the detection window
assert 'class="booth-nav-home"' in out
assert out.index("booth-nav-home") > out.index("<h1>REPORT</h1>") # chip appended after content
def test_verbatim_booth_wrapped_with_back_chip(client):
c, data = client
d = data / "brief"
d.mkdir()
(d / "index.html").write_text("<html><head></head><body><h1>BRIEF</h1></body></html>")
r = c.get("/b/brief/")
assert r.status_code == 200
assert "BRIEF" in r.text # content preserved
assert 'class="booth-nav-home"' in r.text # back chip injected
assert 'href="/"' in r.text
assert 'rel="icon"' in r.text # favicon inherited
def test_verbatim_index_raw_file_route_unwrapped(client):
# the file route (/b/<name>/index.html) still serves the raw bytes — the chip
# only rides on the booth view (/b/<name>/), so downloads/assets stay verbatim
c, data = client
d = data / "brief"
d.mkdir()
(d / "index.html").write_text("<html><body><h1>BRIEF</h1></body></html>")
r = c.get("/b/brief/index.html")
assert r.status_code == 200
assert "booth-nav-home" not in r.text
# ---- .md / .txt in-booth doc viewer -----------------------------------------
def test_doc_kind():
assert doc_kind("notes.md") == "markdown"
assert doc_kind("a.markdown") == "markdown"
assert doc_kind("log.txt") == "text"
assert doc_kind("run.log") == "text"
assert doc_kind("shot.png") is None
assert doc_kind("data.bin") is None
def test_render_doc_markdown():
html, is_html = render_doc("# Title\n\n- a\n- b\n", "markdown")
assert is_html is True
assert "<h1>" in html and "Title" in html
assert "<li>" in html
def test_render_doc_text_is_verbatim():
body, is_html = render_doc("plain\ntext", "text")
assert is_html is False and body == "plain\ntext"
def test_view_markdown_renders(client):
c, data = client
d = data / "run1"; d.mkdir()
(d / "notes.md").write_text("# Heading\n\nsome **bold** text\n")
r = c.get("/b/run1/view", params={"f": "notes.md"})
assert r.status_code == 200
assert "<h1>" in r.text and "Heading" in r.text # rendered, not raw markdown
assert "<strong>bold</strong>" in r.text
assert "attachment" not in r.headers.get("content-disposition", "") # viewed, not downloaded
def test_view_text_shows_preformatted(client):
c, data = client
d = data / "run1"; d.mkdir()
(d / "out.txt").write_text("line one\nline two")
r = c.get("/b/run1/view", params={"f": "out.txt"})
assert r.status_code == 200
assert "<pre" in r.text and "line one" in r.text
assert "attachment" not in r.headers.get("content-disposition", "")
def test_gallery_links_docs_to_view(client):
c, data = client
d = data / "run1"; d.mkdir()
(d / "readme.md").write_text("# hi")
(d / "notes.txt").write_text("hello") # loose txt (no media partner) -> own item
page = c.get("/b/run1/")
assert "view?f=readme.md" in page.text # md -> viewer
assert "view?f=notes.txt" in page.text # txt -> viewer
assert 'href="readme.md" download' not in page.text # not a forced download
# ---- image viewer prev/next nav ---------------------------------------------
def test_view_image_prev_next_nav(client):
c, data = client
d = data / "run1"; d.mkdir()
for n in ("a.png", "b.png", "c.png"):
_touch(d / n)
r = c.get("/b/run1/view", params={"f": "b.png"}) # middle -> prev=a, next=c
assert r.status_code == 200
assert 'class="vnav vprev" href="?f=a.png"' in r.text
assert 'class="vnav vnext" href="?f=c.png"' in r.text
def test_view_image_nav_wraps(client):
c, data = client
d = data / "run1"; d.mkdir()
for n in ("a.png", "b.png", "c.png"):
_touch(d / n)
first = c.get("/b/run1/view", params={"f": "a.png"}).text
assert 'vprev" href="?f=c.png"' in first and 'vnext" href="?f=b.png"' in first # first wraps prev->last
last = c.get("/b/run1/view", params={"f": "c.png"}).text
assert 'vnext" href="?f=a.png"' in last and 'vprev" href="?f=b.png"' in last # last wraps next->first
def test_view_single_image_no_nav(client):
c, data = client
d = data / "run1"; d.mkdir()
_touch(d / "only.png")
r = c.get("/b/run1/view", params={"f": "only.png"})
assert r.status_code == 200
# no arrow anchors with a single image (the .vnav CSS rule is always present)
assert 'class="vnav vprev"' not in r.text and 'class="vnav vnext"' not in r.text
+18
View File
@@ -257,6 +257,24 @@ model_list:
model_info:
mode: rerank
# --- coder-fast → Qwen2.5-Coder-1.5B (BASE), FIM code-completion seat (ana-ml2
# GPU1 :8020, vLLM; deep-research pick 2026-07-27). For Zed editor inline
# edit-predictions via the LEGACY /v1/completions endpoint with Qwen FIM
# markers (<|fim_prefix|>/<|fim_suffix|>/<|fim_middle|>). BASE not -Instruct
# (FIM is a pretraining objective; base completions are cleaner). Apache-2.0.
# mode: completion — this is text-completion, not chat. Reached KEYLESS from
# Vuong's Mac via the zed-fim-proxy (separate port on ana-docker) which injects
# a coder-fast-scoped virtual key; the proxy's model-allowlist + the scoped key
# bound the blast radius. Runner-up was Qwen2.5-Coder-3B (higher HumanEval-FIM,
# non-commercial Qwen-Research license). ---
- model_name: coder-fast
litellm_params:
model: hosted_vllm/qwen2.5-coder-1.5b
api_base: http://10.250.50.54:8020/v1
api_key: os.environ/VLLM_API_KEY
model_info:
mode: completion
# --- z.ai GLM (cloud API) — fronted for unified logging across local
# + cloud inference. Explicit entries, so they win over the "*"
# wildcard below (no collision with llama-swap's glm4.7-flash etc.
+33
View File
@@ -0,0 +1,33 @@
# mimir-inbox runtime config — copy to .env (gitignored on server) and fill the key.
# The 7 RUNTIME vars ONLY. The two DEV-ONLY drift-guard vars
# (WORLDTREE_CONFIG_PATH, MUNINN_GATE_REPO) MUST NOT be set here — they belong
# only to the test environment. All are validated at boot; any failure aborts
# startup with a secret-redacted message naming the variable.
# muninn-gate front door — the co-located gate on corviduo-dev (host network).
MUNINN_GATE_URL=http://10.250.50.152:8090
# infra-ops-provisioned bearer (read+submit+control); SERVER-side only, never logged.
MUNINN_GATE_BEARER_KEY=__FILL_ON_SERVER__
# Shared staging root — RATIFIED value (not an example). Must exist, be a dir, and
# be writable by uid 1000; canonicalized at boot.
MIMIR_INBOX_STAGING_ROOT=/mnt/muninn-staging/mimir-inbox
# The RETRY WINDOW (not a tidiness setting): staged files are retained until the
# job is terminal + this many days. Shortening it shortens retryability
# (INV-MI-19 / worldtree-dev open-in-place). Do not tune down without telling
# mimir-dev.
MIMIR_INBOX_STAGING_RETENTION_DAYS=7
# Wing allowlist — name:description pairs, ";"-separated (first ":" splits
# name/description; a name is one path component; order = select order). Names
# MUST be upstream Worldtree library.wings ids. Boot-fails if empty.
MIMIR_INBOX_WINGS=fiction:Novels, novellas and short stories;main:General knowledge — non-fiction, manuals and technical reference
# Upload ceiling in MiB, enforced mid-stream.
MIMIR_INBOX_MAX_UPLOAD_MB=512
# Unclean-terminal float window (days) — how long a failed / complete_with_warnings
# job sorts above clean ones before settling into chronological order.
MIMIR_INBOX_FLOAT_WINDOW_DAYS=7
+37
View File
@@ -0,0 +1,37 @@
# mimir-inbox — UI front end for large-document / book ingestion into the Muninn
# KB, over the muninn-gate API (browser -> mimir-inbox -> staging -> path-addressed
# POST /jobs to the gate; bytes never reach the gate). No Dockerfile upstream
# (vh/mimir-inbox); this is infra-ops's, modeled on the muninn-gate one.
#
# All 7 RUNTIME deps are public PyPI — NO private index, NO BuildKit secret
# (unlike muninn-gate, which needed muninn-dispatch at runtime).
#
# docker build -t mimir-inbox:0.0.1 .
FROM python:3.11-slim
RUN pip install --no-cache-dir uv==0.11.*
WORKDIR /app
COPY pyproject.toml uv.lock README.md ./
COPY src ./src
# Install project + deps from the lockfile, EXCLUDING the dev group. --no-dev is
# load-bearing, not just leaner: the dev group carries muninn-dispatch via a path
# source ("../Worldtree/packages/muninn-dispatch") that is absent in the image,
# AND INV-MI-7 asserts runtime code never imports muninn_dispatch. Runtime deps
# resolve entirely from PyPI.
RUN uv sync --no-dev --frozen --no-cache
# SINGLE-STAGE BY DESIGN — DO NOT convert to multi-stage that copies only .venv.
# `uv sync` installs the project EDITABLE-linked back to /app/src/mimir_inbox (not
# copied into site-packages), so mimir_inbox.__file__ — and the 16 templates + 442
# static assets (vendored design CSS + htmx) it serves from — resolve out of src/.
# Dropping src/ in a final layer would kill the app at import; if it somehow got
# past import it would 404 every asset. src/ MUST remain alongside .venv.
# (mimir-dev preflight, 2026-08-01: `uv sync --no-dev --frozen` boots green with
# the path source absent; the alternative safe shape is `uv build --wheel` +
# install the wheel, which copies assets into site-packages.)
ENV PATH="/app/.venv/bin:$PATH"
# The launcher (compose) owns --host/--port; no CMD here. Runs as uid 1000 (set
# in compose) — the sole writer to the staging root; /app is read-only to it.
+76
View File
@@ -0,0 +1,76 @@
# mimir-inbox
UI front end for large-document / novel / book ingestion into the Muninn KB —
server-side FastAPI + HTMX over the **muninn-gate** API (#377). The submit path
is: **browser → mimir-inbox `/upload` (writes to staging) → path-addressed
`POST /jobs` to the gate**. Bytes never reach the gate; the gate references the
staged file by absolute path, and the watcher opens it **in place** at claim.
- **Upstream:** `git@gitea.phasefinal.com:vh/mimir-inbox.git` (infra-ops owns
containerization — no Dockerfile upstream).
- **Host:** corviduo-dev `10.250.50.152:8091`, co-located with muninn-gate
(:8090) and the worldtree-personal muninn watcher. Operator ruled co-location
2026-08-01 (reversing the earlier off-box/NFS plan); worldtree-dev approved
the box placement.
- **Posture:** LAN/WG-internal, no TLS, no public exposure (same as the gate).
No datastore, no migrations, no writable path other than the staging root — a
restart loses nothing.
## Why co-located (not off-box/NFS)
The watcher opens the staged file **in place** at claim (`core/muninn/runner.py`),
so staging's only hard constraint is **path identity** — the writer, gate, and
watcher must resolve the same absolute string to the same bytes. Co-location on
corviduo-dev buys that outright with one local filesystem and sidesteps the NFS
failure modes (path-identity drift, TOCTOU widening, stale handles, a hung mount
blocking the async staging check on mimir-inbox's event loop). Multi-host stays
viable later behind a same-path shared mount, but co-location is the cheapest
way to path identity.
## Retention rule (load-bearing — INV-MI-19)
Because the watcher opens the file in place, **a staged file must persist from
submit until the job is terminal** (complete, or failed-and-not-retried) — retry
re-runs the structure phase, which re-opens the same staged path. Deleting on
201-submit would kill every retried job at claim with a `not_found` that looks
exactly like a namespace mismatch. `MIMIR_INBOX_STAGING_RETENTION_DAYS` is the
**retry window**, not a tidiness knob; do not tune it down without telling
mimir-dev.
## Config
All runtime config is environment variables — no config file. The real values
(including the bearer-key secret) live in a gitignored `.env` next to
`compose.yaml` on the server (`0600`); the repo carries `.env.example`. Two
DEV-ONLY drift-guard vars (`WORLDTREE_CONFIG_PATH`, `MUNINN_GATE_REPO`) must
**not** be set in the container.
## Deploy
```bash
# 1. Fetch the source at the pinned tag/HEAD onto corviduo-dev, build the image
# (all deps public PyPI; uv sync --no-dev is load-bearing — see Dockerfile):
docker build -t mimir-inbox:0.0.1 .
# 2. Provision the shared staging root (mimir-inbox-writable, gate+watcher :ro):
sudo install -d -o 1000 -g 1000 -m 0755 /mnt/muninn-staging/mimir-inbox
# 3. Drop the real .env (0600) next to compose.yaml with the bearer key filled
# from the infra-ops key drop (~/.config/muninn-gate/mimir-inbox-env.sh).
# 4. Bring it up:
docker compose up -d
# 5. muninn-dev's one-file path-agreement probe, then the acceptance run.
```
**Operational guard:** docker fabricates a MISSING bind source as an empty dir
that passes checks — confirm `/mnt/muninn-staging/mimir-inbox` exists on the host
before `up -d` (step 2 does this). The container also fails boot if the staging
root is not writable (`os.access(W_OK)`), so a perms miss surfaces loudly.
## Health
`GET /health-banner` is the HTMX health surface (reflects gate reachability).
The container healthcheck is a **TCP-connect liveness** probe only — it must not
couple to the gate's state, so a gate outage never restarts mimir-inbox.
+59
View File
@@ -0,0 +1,59 @@
# mimir-inbox — large-document / book ingestion UI over the muninn-gate API (#377).
# Deployed on corviduo-dev (10.250.50.152), co-located with muninn-gate (:8090)
# and the worldtree-personal muninn watcher. Operator ruled co-location 2026-08-01
# (reversing the earlier off-box/NFS plan); worldtree-dev approved the box
# placement. infra-ops-managed stack, separate from the worldtree CI/CD compose.
#
# Image built out-of-band (all deps public PyPI, no secret needed):
# docker build -t mimir-inbox:0.0.1 .
#
# Runtime config (incl. the bearer-key secret) lives on the server in a gitignored
# .env next to this compose; repo carries the redacted .env.example.
services:
mimir-inbox:
image: mimir-inbox:0.0.1
container_name: mimir-inbox
restart: unless-stopped
# uid 1000 (vh) — the SOLE writer to the staging root (INV-MI-3), which is
# provisioned 1000:1000. Matches the corviduo-dev muninn stack (gate is 1000).
user: "1000:1000"
# Host networking: the WG-internal bind lands directly on corviduo-dev, and
# MUNINN_GATE_URL=http://10.250.50.152:8090 reaches the co-located gate (whose
# own bind is host-network on that address).
network_mode: host
# The 7 runtime vars incl. the secret. The two DEV-ONLY drift-guard vars
# (WORLDTREE_CONFIG_PATH, MUNINN_GATE_REPO) are deliberately absent — they
# must not be set in the container.
env_file:
- .env
command:
- uvicorn
- --factory
- mimir_inbox.app:create_app
- --host
- "10.250.50.152"
- --port
- "8091"
volumes:
# Shared staging root — mimir-inbox is the SOLE WRITER; the gate + watcher
# bind this SAME absolute path :ro. A corviduo-dev-LOCAL dir (path identity
# is the real constraint; the watcher opens staged files IN PLACE at claim
# — worldtree-dev, core/muninn/runner.py:362-367 — so co-location buys path
# identity outright and nothing crosses a device boundary).
- /mnt/muninn-staging/mimir-inbox:/mnt/muninn-staging/mimir-inbox:rw
healthcheck:
# LIVENESS ONLY — a TCP connect proves uvicorn is bound, WITHOUT coupling
# container health to gate reachability. A gate outage must NOT restart
# mimir-inbox; the health banner surfaces that state instead (same lesson
# as the gate's /ping-not-/health probe).
test: ["CMD", "python", "-c", "import socket; socket.create_connection(('10.250.50.152',8091),3).close()"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
labels:
- homepage.group=Worldtree
- homepage.name=Mimir Inbox
- homepage.icon=mdi-book-arrow-right
- homepage.description=Large-document ingestion UI for the Muninn KB (#377)
- homepage.href=http://10.250.50.152:8091/
+30
View File
@@ -0,0 +1,30 @@
# muninn-gate — WG-internal HTTP front door for the Muninn ingestion queue (#377).
# No Dockerfile in the upstream repo (vh/muninn-gate); this is infra-ops's.
# muninn-dispatch is pinned in pyproject (currently ==0.1.5, from vh/muninn-gate
# main; bumped from 0.1.4 for the concept_schema/concept_schema_source fields)
# and comes from the internal Gitea PyPI index "gitea"; the read token (vh) is
# passed as a BuildKit secret (never baked into a layer). Also tag the image with
# the source SHA for traceability, since the gate version (0.0.14) is unchanged.
#
# DOCKER_BUILDKIT=1 docker build \
# --secret id=gitea_pw,src=<vh-token-file> \
# -t muninn-gate:0.0.14 -t muninn-gate:<src-sha> .
FROM python:3.11-slim
RUN pip install --no-cache-dir uv==0.11.* \
&& useradd --system --uid 10014 --create-home --home-dir /home/gate gate
WORKDIR /app
COPY pyproject.toml README.md uv.lock ./
COPY src ./src
# Install the app + deps. muninn-dispatch resolves only from the "gitea" index
# (pinned via [tool.uv.sources]); everything else from PyPI.
RUN --mount=type=secret,id=gitea_pw \
UV_INDEX_GITEA_USERNAME=vh \
UV_INDEX_GITEA_PASSWORD="$(cat /run/secrets/gitea_pw)" \
uv pip install --system --no-cache .
# The launcher (compose) owns the bind — no CMD here. The app carries `bind`
# for observability only; --host/--port passed by compose are authoritative.
USER gate
+56
View File
@@ -0,0 +1,56 @@
# muninn-gate
WG-internal HTTP front door for the Muninn ingestion queue (Worldtree #377).
Path-addressed submit, list/status, cancel/retry, watcher-liveness health.
Upstream: `vh/muninn-gate` (no Dockerfile there — this stack owns containerization).
## Where it runs
**corviduo-dev (10.250.50.152)**, co-located with the worldtree-personal muninn
watcher (`worldtree-personal-worldtree-muninn-1`). infra-ops-managed stack,
separate from the worldtree CI/CD compose. Bind: `10.250.50.152:8090`, `network_mode: host`.
## Key wiring (why it's shaped this way)
- **`ingestion_root: /data/state/ingestion`** — the `worldtree-personal_worldtree-state`
volume, mounted at `/data/state`. Byte-identical path to the watcher's view; the
dispatch records absolute paths and both processes read each other's.
Acceptance: `/health``watcher.running: true` **proves** this byte-identity
(gate is reading the heartbeat the watcher writes). `no_heartbeat` with the
watcher up = root mismatch.
- **`user: "1000:1000"`** — the ingestion dir is `vh:vh 0755`; the gate must run as
uid 1000 to write the queue (non-root, least-privilege). Overrides the image user.
- **staging** — `/mnt/muninn-staging/mimir-inbox`, same absolute path bound `:ro` in
BOTH the gate and the watcher (worldtree-dev's b162). Currently a LOCAL placeholder
dir; becomes the shared mount when mimir-inbox (the writer) lands. **No gate-side
check can tell a real share from an empty dir** (docker fabricates a missing bind
source) — the guard is operational: confirm the host mount exists before wiring/
repointing a bind. Repoint needs a restart (config is read once at boot).
- **health probe uses `/ping`, not `/health`** — `/health` is always-200 by design
(watcher-down is report *content*, not a status code), so it must never be the
liveness probe or it would restart the wrong container.
## Config (single-writer, infra-ops)
Real config with bearer-key secrets lives on the server at
`/opt/docker/conf/muninn-gate/muninn-gate.yaml` (`1000:1000`, `0600`, gitignored).
Redacted schema: `conf/muninn-gate.example.yaml`. Schema is CLOSED — any unknown
field is a boot failure. Keys are flat-scope (`read | submit | control`, no
inheritance); `name` is the non-secret `submitted_by` identity.
## Build + deploy
Image builds out-of-band — the Gitea read token (for `muninn-dispatch==0.1.4` from
the internal index) rides as a BuildKit secret, never a layer:
```bash
DOCKER_BUILDKIT=1 docker build --secret id=gitea_pw,src=<gitea-read-token> -t muninn-gate:0.0.14 .
docker compose up -d # after installing the config to /opt/docker/conf/muninn-gate/
```
## Status
Booted + healthy 2026-07-31; `/ping` + `/health` (watcher:running:true,
ingestion_root_writable:true) verified. **SUBMIT deferred** — returns `not_found`
against the placeholder staging until the real shared mount + mimir-inbox writer
exist; then repoint + the one-file path-agreement probe + the real acceptance run.
+70
View File
@@ -0,0 +1,70 @@
# muninn-gate — WG-internal HTTP front door for the Muninn ingestion queue (#377).
# Deployed on corviduo-dev (10.250.50.152), co-located with the worldtree-personal
# muninn watcher (worldtree-personal-worldtree-muninn-1). infra-ops-managed stack,
# separate from the worldtree CI/CD compose.
#
# Image is built out-of-band (the Gitea read token rides as a BuildKit secret, so
# it never lands in a layer) — see README.md:
# DOCKER_BUILDKIT=1 docker build --secret id=gitea_pw,src=<token> -t muninn-gate:0.0.14 .
#
# Real config (with bearer-key secrets) lives on the server at
# /opt/docker/conf/muninn-gate/muninn-gate.yaml (gitignored); repo carries the
# redacted conf/muninn-gate.example.yaml.
services:
muninn-gate:
image: muninn-gate:0.0.14
container_name: muninn-gate
restart: unless-stopped
# Run as the ingestion-owner uid (vh, 1000:1000) so the gate can WRITE the
# queue (ingestion dir is 1000:1000 0755). Overrides the image's build user;
# non-root, least-privilege (no root needed — binds :8090, reads config +
# staging, writes only ingestion). The mounted config is chowned 1000:1000 0600.
user: "1000:1000"
# WG-internal bind straight to the host WG address; the launcher owns the bind
# (the app's `bind` field is observability-only). Host networking so --host/--port
# land directly on corviduo-dev.
network_mode: host
command:
- uvicorn
- muninn_gate.entrypoint:app_factory
- --factory
- --host
- "10.250.50.152"
- --port
- "8090"
environment:
MUNINN_GATE_CONFIG: /app/config/muninn-gate.yaml
volumes:
# ingestion_root = /data/state/ingestion — the SAME volume the personal muninn
# watcher uses, byte-identical path in both containers (absolute-path agreement).
# NOTE (hardening candidate): whole state volume per muninn-dev's spec; a subpath
# mount of just `ingestion` -> /data/state/ingestion would be tighter (gate only
# needs RW on ingestion). Flag to muninn-dev before adopting.
- worldtree-personal_worldtree-state:/data/state
# single-writer mounted config (bearer-key secrets), read-only:
- /opt/docker/conf/muninn-gate/muninn-gate.yaml:/app/config/muninn-gate.yaml:ro
# shared staging root — SAME absolute path as the watcher's bind, read-only
# (gate only resolves + reads staging; never writes there):
- /mnt/muninn-staging/mimir-inbox:/mnt/muninn-staging/mimir-inbox:ro
healthcheck:
# /ping is the one anonymous route; returns {"service":"ok"}. NB /health must
# NOT be the liveness probe — it is always-200 by design even when the watcher
# is down, so it would never restart the gate (it would wrongly signal the
# watcher's container).
test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if b'ok' in urllib.request.urlopen('http://10.250.50.152:8090/ping', timeout=3).read() else 1)"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
labels:
- homepage.group=Worldtree
- homepage.name=Muninn Gate
- homepage.icon=mdi-gate-arrow-right
- homepage.description=Muninn ingestion queue front door (#377)
- homepage.href=http://10.250.50.152:8090/ping
volumes:
# The worldtree-personal muninn watcher's state volume — external, managed by the
# worldtree-personal compose. We attach to it read/write for the ingestion queue.
worldtree-personal_worldtree-state:
external: true
@@ -0,0 +1,34 @@
# muninn-gate mounted config — REDACTED EXAMPLE (committed).
# The real file (with bearer-key secrets) lives on corviduo-dev at
# /opt/docker/conf/muninn-gate/muninn-gate.yaml and is gitignored.
#
# Schema is CLOSED: an unknown field (top-level or per-key) is a BOOT FAILURE,
# not a warning. Read once at boot; every validation failure aborts before the
# socket binds. Four top-level fields only.
# Absolute, existing directory, same volume as the personal muninn watcher.
# Byte-identical to the watcher's view: /data/state/ingestion in both containers.
ingestion_root: /data/state/ingestion
# Submit-path allowlist (non-empty; each absolute, existing, a directory).
# Canonicalized at boot. Ratified 2026-07-30. Real shared dir as of 2026-08-01
# (mimir-inbox — the writer — deployed on corviduo-dev:8091): 1000:1000 0755,
# writer rw / gate+watcher ro. Path-agreement probe PASS (muninn-dev, same
# inode+bytes across writer/gate/watcher; admission byte-identity preserving).
staging_roots:
- /mnt/muninn-staging/mimir-inbox
# Bearer credentials. `name` = non-secret caller identity, recorded as
# `submitted_by`. Flat scopes (no hierarchy/inheritance): read | submit | control.
# Names AND key values must both be unique; a padded key fails boot.
keys:
- name: mimir-inbox
key: <64-hex-secret, provisioned by infra-ops>
scopes: [read, submit, control] # control added 2026-08-01 for mimir-dev cancel/retry
- name: ops-curl
key: <different 64-hex-secret>
scopes: [read, submit, control]
# WG-internal address — observability only, NOT read by the app. Keep consistent
# with the launcher's --host/--port (the launcher is authoritative).
bind: 10.250.50.152:8090
+25 -6
View File
@@ -98,7 +98,10 @@ GRANITE_SERVED_NAME=granite-4.1-8b
# short parallel calls, so the 64K cap is ample.
# 131072 — RESTORED 2026-07-16 (native max) for full-chapter summarization; GPU-1
# freed by the image-bench evict + qwen36-vl gone, so the 128K ctx fits again.
GRANITE_MAX_MODEL_LEN=131072
# 16384 — SHRUNK 2026-07-27 (operator: granite is being phased out) to free GPU-1
# room for vllm-coder (the Zed FIM seat). Full-chapter ctx dropped; util 0.13 holds
# 16K cleanly (32768 @ util 0.12 crash-looped: KV est-max was only 19376 tokens).
GRANITE_MAX_MODEL_LEN=16384
# FP8 KV cache (native on Blackwell cc 12.0). At 50K ≈ ~4.2 GB (vs ~8.4 GB at fp16).
GRANITE_KV_CACHE_DTYPE=fp8
# util 0.35 (~33.6 GB) — tuned 2026-06-13 to leave ~3.5 GB free on GPU 1 alongside
@@ -118,8 +121,24 @@ GRANITE_KV_CACHE_DTYPE=fp8
# 0.27 — RE-GROWN 2026-07-16 (same session) after char-rp moved onto GPU-1: spend the
# leftover room on full-chapter context (max-len 131072). KV 15.0 GiB = 196,560 tokens
# = 1.50x @ 131072; GPU-1 lands ~6.7 GB headroom (char-rp 30 + granite 27 + selene 17 + trio).
GRANITE_GPU_MEM_UTIL=0.27
# Concurrency cap — set VERY HIGH 2026-07-16 (was vLLM default 128) so the KV pool is
# the only bound. granite is the fleet fan-out summarizer/classifier (many concurrent
# SHORT calls); default 128 capped below the KV bound (~192 @ 1K-tok). VRAM-neutral.
GRANITE_MAX_NUM_SEQS=1024
# 0.13 — SHRUNK 2026-07-27 (was 0.27) with the max-len drop; frees ~14 GB on GPU-1 for
# vllm-coder. granite phasing out, so no longer worth the big KV pool. (0.12 was too
# small for even 16K KV; 0.13 gives ~1.4x @ 16384.)
GRANITE_GPU_MEM_UTIL=0.13
# Concurrency cap. Was 1024 (fan-out summarizer); DROPPED to 256 on 2026-07-27 with the
# phasing-out shrink (256 is ample for the reduced summarizer load; smaller sched state).
GRANITE_MAX_NUM_SEQS=256
# Qwen2.5-Coder-1.5B (BASE) — FIM code-completion seat for Zed editor inline
# edit-predictions (deep-research pick 2026-07-27, Apache-2.0). GPU-1, alongside the
# small-model trio + (phasing-out) granite. Reached via LiteLLM `coder-fast` alias
# and the keyless zed-fim-proxy (stacks/zed-fim-proxy). util 0.06 (~5.7 GB) holds
# the 1.5B fp16 + fp8 KV; 8192 ctx is ample for FIM (KV = 13.75x concurrency).
CODER_PORT=8020
CODER_GPU_ID=1
CODER_MODEL=Qwen/Qwen2.5-Coder-1.5B
CODER_SERVED_NAME=qwen2.5-coder-1.5b
CODER_MAX_MODEL_LEN=8192
CODER_KV_CACHE_DTYPE=fp8
CODER_GPU_MEM_UTIL=0.06
CODER_MAX_NUM_SEQS=32
+64
View File
@@ -278,6 +278,70 @@ services:
- homepage.description=Granite 4.1 8B FP8 via vLLM (ana-ml2)
- homepage.href=http://10.250.50.54:${GRANITE_PORT}/docs
vllm-coder:
image: vllm/vllm-openai:${VLLM_VERSION}
container_name: vllm-coder
restart: unless-stopped
ipc: host
ports:
- "${CODER_PORT}:8000"
volumes:
- /tank/aimodels/huggingface:/hfcache
environment:
- HF_HOME=/hfcache
- HF_HUB_CACHE=/hfcache/hub
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
- VLLM_API_KEY=${API_KEY:-}
command:
# Qwen2.5-Coder-1.5B (BASE) — FIM code-completion seat for Zed edit-predictions
# (deep-research pick 2026-07-27). Native fill-in-the-middle: <|fim_prefix|> /
# <|fim_suffix|> / <|fim_middle|> (IDs 151659/151660/151661); Zed sends the
# FIM-formatted prompt to /v1/completions and vLLM passes it through (the FIM
# special tokens live in the tokenizer). BASE not -Instruct (FIM is a
# pretraining objective; base completions are cleaner). Apache-2.0. Runner-up =
# Qwen2.5-Coder-3B (higher HumanEval-FIM but non-commercial Qwen-Research license).
- ${CODER_MODEL}
- --served-model-name
- ${CODER_SERVED_NAME}
- --host
- 0.0.0.0
- --port
- "8000"
- --gpu-memory-utilization
- ${CODER_GPU_MEM_UTIL}
- --max-model-len
- ${CODER_MAX_MODEL_LEN}
- --max-num-seqs
- ${CODER_MAX_NUM_SEQS}
- --dtype
- auto
- --kv-cache-dtype
- ${CODER_KV_CACHE_DTYPE}
- --enable-prefix-caching
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids:
- "${CODER_GPU_ID}"
capabilities:
- gpu
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 300s
networks:
- tnet
labels:
- homepage.group=AI - Inference
- homepage.name=vLLM Qwen2.5-Coder 1.5B (FIM)
- homepage.icon=mdi-code-braces
- homepage.description=Qwen2.5-Coder-1.5B FIM code-completion (ana-ml2, Zed edit-predictions)
- homepage.href=http://10.250.50.54:${CODER_PORT}/docs
networks:
tnet:
name: traefik-net
+23
View File
@@ -0,0 +1,23 @@
# zed-fim-proxy tunables — copy to `.env` on ana-docker (the real .env holds the
# scoped key and is server-only / gitignored). See conf/proxy.py + README.md.
# Port the keyless route listens on (host network).
ZED_PORT=4141
# LiteLLM gateway to forward to (localhost:4000 via network_mode: host).
ZED_UPSTREAM=http://localhost:4000
# The ONLY model this route will forward (proxy rejects any other "model").
ZED_ALLOWED_MODEL=coder-fast
# Source-IP allowlist (comma-separated, EXACT IPs). Empty = allow all.
# LEFT EMPTY BY DESIGN (operator direction 2026-07-27): Zed roams the operator's
# WireGuard 10.0.0.0/8, so a single-IP pin would break it — do NOT tighten. The
# route is bounded instead by the coder-fast-scoped key + the model/path allowlist
# (keyless but coder-fast-only, internal-net-only). NOTE: this proxy matches exact
# IPs only — scoping to a CIDR like 10.0.0.0/8 would need CIDR support (not added).
ZED_ALLOWED_IPS=
# A LiteLLM virtual key SCOPED TO ZED_ALLOWED_MODEL ONLY (the real blast-radius
# bound). Mint: POST /key/generate {"models":["coder-fast"]}. NEVER commit the value.
ZED_SCOPED_KEY=
+48
View File
@@ -0,0 +1,48 @@
# zed-fim-proxy
A **keyless `/v1/completions` front door** on ana-docker for Zed's editor
edit-prediction (inline FIM completion) feature, which **cannot send an
`Authorization` header**. Runs on a separate port from LiteLLM and forwards to it
with an injected, model-scoped key.
- **Host:** ana-docker `10.250.50.70`, port **4141** (`network_mode: host`).
- **Backs:** the `coder-fast` model (Qwen2.5-Coder-1.5B FIM seat, `stacks/vllm`
`vllm-coder` on ana-ml2:8020) via LiteLLM `:4000`.
- **Zed config** (`edit_predictions.open_ai_compatible_api`): `api_url:
http://10.250.50.70:4141/v1`, `model: coder-fast`, `prompt_format: qwen`,
`max_output_tokens: <n>`. (The proxy also accepts `http://10.250.50.70:4141` —
it matches both `/v1/completions` and `/completions`.)
## Security model (stdlib proxy in `conf/proxy.py`)
Four guards + a scoped key — a keyless route that injects a working key is only
safe if it can't be pivoted:
1. **POST + path** `/v1/completions` (or `/completions`) only. `GET /ping` is an
anonymous liveness (`{"service":"ok"}`). `/v1/chat/completions` is rejected.
2. **Model allowlist** — request body `model` must equal `ZED_ALLOWED_MODEL`
(`coder-fast`); anything else → 403.
3. **Injected scoped key** — a LiteLLM virtual key scoped to `coder-fast` ONLY
(`POST /key/generate {"models":["coder-fast"]}`). Even if guards 12 were
bypassed, the key reaches nothing else (verified: 403 on `gen`). **This is the
real blast-radius bound.**
4. **Source-IP allowlist** (`ZED_ALLOWED_IPS`) — **intentionally left OFF**
(operator direction 2026-07-27): Zed roams the operator's WireGuard
`10.0.0.0/8`, so a single-IP pin would break it. **Do NOT tighten.** The
keyless route is bounded by guards 13 (coder-fast-only, `/v1/completions`-only,
scoped key) and is internal-network only. (The proxy matches exact IPs; a
`10.0.0.0/8` CIDR would need CIDR support — deliberately not added.)
## Deploy
```
# conf/proxy.py -> /opt/docker/conf/zed-fim-proxy/proxy.py
# compose.yaml -> /opt/docker/compose/zed-fim-proxy/compose.yaml
# .env (from .env.example, with ZED_SCOPED_KEY filled) -> same dir, mode 600
cd /opt/docker/compose/zed-fim-proxy && docker compose up -d
# verify keyless:
curl -s http://localhost:4141/v1/completions -H 'Content-Type: application/json' \
-d '{"model":"coder-fast","prompt":"def add(a,b):\n return","max_tokens":16,"temperature":0.2}'
```
Stdlib-only proxy (no pip) in a bare `python:3.12-slim` container — no build.
+36
View File
@@ -0,0 +1,36 @@
name: zed-fim-proxy
# Keyless /v1/completions front door for Zed edit-predictions (coder-fast only).
# See conf/proxy.py for the guard model. Runs network_mode: host so it (a) sees the
# real client source IP for the best-effort allowlist (docker port-publish would NAT
# it away) and (b) reaches the LiteLLM gateway on localhost:4000. Stdlib-only proxy
# in a bare python image — no build, no pip.
services:
zed-fim-proxy:
image: python:3.12-slim
container_name: zed-fim-proxy
restart: unless-stopped
network_mode: host
command: ["python", "/app/proxy.py"]
volumes:
- /opt/docker/conf/zed-fim-proxy/proxy.py:/app/proxy.py:ro
environment:
- ZED_PORT=${ZED_PORT:-4141}
- ZED_UPSTREAM=${ZED_UPSTREAM:-http://localhost:4000}
- ZED_ALLOWED_MODEL=${ZED_ALLOWED_MODEL:-coder-fast}
# best-effort source-IP allowlist (comma-separated); empty = allow all.
- ZED_ALLOWED_IPS=${ZED_ALLOWED_IPS:-}
- ZED_SCOPED_KEY=${ZED_SCOPED_KEY}
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request,os; urllib.request.urlopen('http://localhost:%s/ping' % os.environ.get('ZED_PORT','4141'))"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
labels:
- homepage.group=AI - Gateways & Chat
- homepage.name=Zed FIM Proxy
- homepage.icon=mdi-code-braces-box
- homepage.description=Keyless /v1/completions for Zed edit-predictions (coder-fast, ana-docker)
- homepage.href=http://10.250.50.70:4141/ping
+102
View File
@@ -0,0 +1,102 @@
"""zed-fim-proxy — a keyless front door for Zed's edit-prediction feature.
Zed's edit_predictions.open_ai_compatible_api provider cannot send an
Authorization header, so it needs a route where POST /v1/completions succeeds
with no key. This proxy is that route, on a separate port from LiteLLM, with
four narrow guards + an injected model-scoped key:
1. POST only, path exactly /v1/completions (GET /ping is an anonymous liveness).
2. source-IP allowlist (best-effort — set ZED_ALLOWED_IPS; empty = allow all).
NOTE: only meaningful if this process sees the real client IP — run the
container with network_mode: host (docker port-publish would NAT the source
to the bridge gateway and defeat it). Behind a site-to-site NAT it may still
see a mesh IP, not the Mac — verify against the access log at deploy.
3. request body "model" must equal ZED_ALLOWED_MODEL (default coder-fast).
4. injects Authorization: Bearer $ZED_SCOPED_KEY (a LiteLLM virtual key scoped
to that ONE model) and forwards to $ZED_UPSTREAM. The scoped key is the real
blast-radius bound: even if guards 1-3 were bypassed, the key can reach
nothing but coder-fast.
Stdlib only (no pip) — runs in a bare python:slim container.
"""
import json
import os
import sys
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
UPSTREAM = os.environ.get("ZED_UPSTREAM", "http://localhost:4000").rstrip("/")
SCOPED_KEY = os.environ["ZED_SCOPED_KEY"]
ALLOWED_MODEL = os.environ.get("ZED_ALLOWED_MODEL", "coder-fast")
ALLOWED_IPS = set(x.strip() for x in os.environ.get("ZED_ALLOWED_IPS", "").split(",") if x.strip())
PORT = int(os.environ.get("ZED_PORT", "4141"))
TIMEOUT = float(os.environ.get("ZED_TIMEOUT", "60"))
class Handler(BaseHTTPRequestHandler):
server_version = "zed-fim-proxy/1.0"
def _json(self, code, obj):
body = json.dumps(obj).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_GET(self):
if self.path.rstrip("/") == "/ping":
return self._json(200, {"service": "ok"})
return self._json(404, {"error": "not found"})
def do_POST(self):
src = self.client_address[0]
if ALLOWED_IPS and src not in ALLOWED_IPS:
return self._json(403, {"error": f"source {src} not allowed"})
# Accept both /v1/completions and /completions so the Zed api_url can be
# set to either http://host:4141/v1 or http://host:4141 (Zed appends
# /completions). /v1/chat/completions is NOT in the set → stays rejected.
if self.path.split("?")[0].rstrip("/") not in ("/v1/completions", "/completions"):
return self._json(404, {"error": "only POST /v1/completions or /completions"})
try:
length = int(self.headers.get("Content-Length", 0))
except ValueError:
return self._json(400, {"error": "bad content-length"})
raw = self.rfile.read(length)
try:
body = json.loads(raw)
except Exception:
return self._json(400, {"error": "invalid json body"})
if body.get("model") != ALLOWED_MODEL:
return self._json(403, {"error": f"model must be '{ALLOWED_MODEL}'"})
req = urllib.request.Request(
UPSTREAM + "/v1/completions",
data=raw,
method="POST",
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {SCOPED_KEY}",
},
)
try:
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
data, code, ctype = r.read(), r.status, r.headers.get("Content-Type", "application/json")
except urllib.error.HTTPError as e:
data, code, ctype = e.read(), e.code, e.headers.get("Content-Type", "application/json")
except Exception as e: # noqa: BLE001
return self._json(502, {"error": f"upstream error: {e}"})
self.send_response(code)
self.send_header("Content-Type", ctype)
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def log_message(self, fmt, *args):
sys.stderr.write("%s - %s\n" % (self.client_address[0], fmt % args))
if __name__ == "__main__":
print(f"zed-fim-proxy on :{PORT} -> {UPSTREAM} (model={ALLOWED_MODEL}, "
f"ip-allowlist={'set' if ALLOWED_IPS else 'OFF'})", flush=True)
ThreadingHTTPServer(("0.0.0.0", PORT), Handler).serve_forever()