diff --git a/persistent-memory.md b/persistent-memory.md index 45c5712..84e12a9 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-07-02_ +_Last updated: 2026-07-05_ ## Repo purpose @@ -22,12 +22,13 @@ Sister repos (separate gitea repos, deployed by playbooks here): | `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) | | `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) | | `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) | -| `vh/althing` | Lean trusted inter-agent message bus — **v0.17 multi-machine** (re-architected 2026-06): each box runs a local-SQLite bus + a courier/receiver for P2P over the 10.x net. The v0.15 lean-bus cut RIPPED moderation / chamber (UI 7881) / forseti-daemon / agent-runner / redis-valkey. | per-box `uv tool install` (NOT CI-deploy); **nh3-extdev** added as a mesh peer 2026-06-25 (model B: althing-svc svc account + shared `/srv/althing`) | +| `vh/althing` | Lean trusted inter-agent message bus — **v2 "email model" (v2.0.0b2, 2026-07)**: per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API `/owner/*` / `althing-mcp` stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. | per-box `uv tool install` (NOT CI-deploy); **nh3-dev = the DEV box** (editable install of `~/development/althing`, gets new versions first); **nh3-extdev** a mesh peer (model B: althing-svc + shared `/srv/althing`) | | `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) | | `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) | -| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker** (the image-bloat source); claude-bot now ADMIN collaborator (2026-06-20) | push-to-main → CI build-and-deploy (runner on ana-docker) | +| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) | | `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` | | `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook | +| `model-training-forge` (mtf-dev) | Fine-tuning recipe forge; **T1 = qwopus E-RP writing LoRA** (SFT→DPO, LitBench-RM reward) | training runs, not a deployed sidecar | (`vh/volva` + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev `.service` units were removed — @@ -55,41 +56,34 @@ no longer deployed sidecars here. See Recent decisions.) auths against **demo only**. Personal-instance admin (the `~/.config/worldtree/personal-admin-token`, mode 600) POSTs `/admin/keys` (mints per-project keys; takes `user_id`+`label`, **no - scope param** — scopes are tier-derived, so admin-tier scopes like - `admin.events.read` must be granted WT-side by worldtree-dev). + scope param** — scopes are tier-derived). **On-instance mint recipe + (cleaner than DB-manip):** `docker exec worldtree-worldtree-api-1` POST + `/admin/keys` with the in-container `WORLDTREE_BOOTSTRAP_ADMIN_KEY`; cleartext + once in `.key`=`wt_live_+16hex`. auto-memory `reference_worldtree_demo_key_mint`. - **Per-project user keys against personal Worldtree** (issued - 2026-05-19): `skaldsong:79744637` (nh3-dev iteration), - `skaldsong:7c1dbbbe` (ana-docker prod), `althing:50d85460`, - `mead-hall:a360822d`. Same `user_id=skaldsong` across both - skaldsong keys → shared Heimdall agent slot; different `key_id` - → independently rotatable. Pattern: mint via `/admin/keys`, drop + 2026-05-19): `skaldsong:79744637`, `skaldsong:7c1dbbbe`, + `althing:50d85460`, `mead-hall:a360822d`. Mint via `/admin/keys`, drop value to `/tmp/wt-personal-.key` mode 600, dev collects + shreds (DO NOT cat to chat transcript). -- **Skaldsong CD pattern (registry-pull).** Differs from althing / - asset-engine which build-on-host. vh/skaldsong's CI builds and +- **Skaldsong CD pattern (registry-pull).** vh/skaldsong's CI builds and pushes `gitea.phasefinal.com/vh/skaldsong:` + `:latest`; `playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates. - SHA-pin only (no `:latest` health-gated advance yet). Prereq: host - needs `docker login gitea.phasefinal.com` once (read:package PAT) — - not currently in the workflow. + SHA-pin only. Prereq: host needs `docker login gitea.phasefinal.com` once. - **gitea internal route for fleet hosts.** gitea is a container on **ana-docker** — git-SSH `10.250.50.70:222`, HTTP `:3000`. Fleet/colo hosts must use this internal route, NOT public `gitea.phasefinal.com` - (`38.120.12.44`, ana-srv1) — the public path fail2bans the host egress - IP and wedges webhook deploys. `:22` on `10.250.50.70` is ana-docker's - HOST sshd, not gitea. Full gotcha in `docs/orientation.md` → Git/gitea. + (`38.120.12.44`) — the public path fail2bans the host egress IP. Full + gotcha in `docs/orientation.md` → Git/gitea. - **docker-as-root pattern** (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): `docker run --rm -v - :/wt [-v /var/run/docker.sock:/var/run/docker.sock] - docker:cli sh -c "..."` (or `alpine` for plain file ops). docker-group - membership is effectively root via bind-mount; treat as sudo-equivalent. - **Foot-gun: relative paths in compose.yaml resolve against the sandbox - CWD but the Docker daemon interprets them against the HOST fs — always - pass `-e VAR=/abs/path` for any relative-default config dir.** + :/wt docker:cli sh -c "..."`. docker-group membership is + effectively root via bind-mount. **Foot-gun: relative paths in compose.yaml + resolve against the sandbox CWD but the daemon interprets them against the + HOST fs — always pass `-e VAR=/abs/path` for any relative-default config dir.** - **`scripts/elway` sudo handling** — elway prompts for the sudo password ONCE via `getpass` before the first `sudo: true` step → can't run @@ -98,247 +92,200 @@ no longer deployed sidecars here. See Recent decisions.) - **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On - ana-docker there are TWO identities: the **default `ssh ana-docker` = - `lkraven`** (docker-group, NO passwordless sudo — docker works, root-file - edits don't); **but `ssh infra-ops@ana-docker` HAS NOPASSWD root** (verified - 2026-06-20). **→ For any sudo op on ana-docker (mount, root-owned files, - service control), use `ssh infra-ops@ana-docker`, NOT the default session.** - lkraven-owned files (litellm config, most stack compose/conf) still take - plain `cp`/edit under either identity. **NEW (2026-06-25):** `ssh infra-ops@10.100.10.50` - (nh3-dev) ALSO has NOPASSWD sudo; on **nh3-extdev** infra-ops is sudo-LESS by - design and **`ssh lkraven@10.100.50.42` is the NOPASSWD path** there. + ana-docker: **default `ssh ana-docker` = `lkraven`** (docker-group, NO + passwordless sudo); **`ssh infra-ops@ana-docker` HAS NOPASSWD root**. **→ + For any sudo op on ana-docker, use `ssh infra-ops@ana-docker`.** `ssh + infra-ops@10.100.10.50` (nh3-dev) ALSO NOPASSWD sudo; on **nh3-extdev** infra-ops + is sudo-LESS by design (`ssh lkraven@10.100.50.42` is the NOPASSWD path). **irv-ml1: + `ssh irv-ml1` = lkraven, docker-group (plain docker) but sudo needs a PASSWORD + (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight -_As of 2026-07-02:_ +_As of 2026-07-05:_ -- **`gen` model = qwopus (`qwen3.5-122-a10b`); the Deckard trial is CONCLUDED.** Trialed - `robbatt/Qwen3.6-40B-Deckard-NVFP4` behind the `gen` aliases — it won writing quality - decisively but lost on speed (~36 vs qwopus's ~90 tok/s); reverted (git `681eb70`). - Deckard kept STAGED on ana-ml2 as T1's quality benchmark - (`/tank/aimodels/qwen36-40b-deckard-{bf16,nvfp4}`; container `vllm-deckard-40b` - stopped-but-kept). Full arc in auto-memory `reference_gen_qwopus_122b`. +- **T1 TRAIN — venue decision live; NEXT ACTION = the ana-ml2 SMOKE.** T1 (qwopus E-RP writing + LoRA) datasets + recipe + smoke set are all staged (mtf-dev, on `nh3-dev:data/export/qwen-3.5-122b-erp-lora/` + incl. `smoke/`); bf16 base is on ana-ml2 `/tank/aimodels/qwopus3.5-122b-a10b-bf16` (233G, PUBLIC + HF repo, zero-auth pull). On-prem RULED OUT except a full ana-ml2 shutdown → CPU-offload → ~1-day + fleet outage; cloud (Vast.ai 8×80GB, ~3-6h, ~$60-500, zero impact) is the clean alt. **Operator + chose smoke-first on ana-ml2** to measure real samples/sec before committing. ⚠️ the smoke ITSELF + needs both GPUs + the ~421G vLLM RAM freed (192G VRAM < 244G base → CPU offload) = a brief full + ana-ml2 serving pause; keeping any serving up drops it into the ~week NVMe regime. Full runbook + + both paths in `reference_t1_cloud_train_plan`. Post-train serve-path (swappable-LoRA-on-NVFP4 + test) still queued: `reference_gen_qwopus_122b`. -- **T1 (mtf-dev qwopus writing-LoRA) IN PROGRESS.** Attention-only LoRA (full-attn q/k/v/o - + the bf16 GDN `in_proj_qkv`/`out_proj`); serve via swappable-LoRA-on-NVFP4 (test QUEUED, - gated on the first T1 adapter) or merge+requant. Trainer harness seam MECHANICALLY - proven via a granite-8b spike (2026-07-02): the pipeline runs end-to-end, but efficacy - is NOT validated (mechanics-only synthetic fixture, by design; a granite adapter - wouldn't transfer to qwopus regardless). Real behaviour-shift efficacy is a T1-run - question. NVFP4 quant-structure + serve-path facts in `reference_gen_qwopus_122b`. +- **LitBench-RM UP (irv-ml1 A6000); comfyui DISPLACED.** Respun the R19 creative-writing reward + (`vllm-litbench-rm` :8202, ~19.6G) for T1's reward ensemble; stopped comfyui to free the A6000 + (operator is sole comfyui consumer + holding image-gen until LitBench done). Score scale + re-verified from nh3-dev (literary 0.95 > flat 0.44 > slop 0.12). **TEARDOWN on "litbench done":** + `ssh irv-ml1 'docker rm -f vllm-litbench-rm && docker start comfyui'` — ⚠️ then watch the mmartial + boot trap (root-owned venv → crash-loop; `chown -R 1000:1000 /comfy/mnt/venv`). `reference_litbench_rm_irv_ml1`. -- **Worldtree #332 scoped-log view + tunnel = STANDING ASSET.** `wt_gateway_logs` view + - `wt_readonly` role on the litellm DB + `wt-db-tunnel` systemd on corviduo-dev, for - worldtree-dev's embed-recall regression-watches. #332 fix verified in prod (15×→1.01× - re-embed). Teardown steps + the IP-pin caveat in auto-memory - `reference_wt_gateway_scoped_log_view`. +- **character-rp role — SHIPPED; one QUEUED spot-check.** Proved per-request `extra_body` + (top_k/repetition_penalty) forwards through the `gen-reasoning` alias to the vLLM sampler (no + gateway cap needed); pre-staged the `character-rp` role in demo+personal bind-mount + `model_roles.yaml` (byte-verified live on b18; caught the cached-registry ordering so the deploy's + own restart activates it). worldtree-dev shipped **#344 (v1.0.0b19)** fixing the durable-agent + override-drop (role now applies the RP profile). QUEUED: ratatoskr-dev pings a turn-timestamp → I + spot-check the gateway spend_logs to confirm the RP sampler params (temp 0.75 / top_p 0.85 / + presence 1.5 / top_k 20 / rep 1.0) reach vLLM. -- **`/books` mounted (transient) on nh3-dev** for a books/corpus ingestion: - `10.0.50.50:/mnt/books` (ESH NAS) → `/mnt/books`, NFSv4 `ro,soft` (soft dodges the ESH - D-state hang). NOT fstab — re-mount via `infra-ops@10.100.10.50` if nh3-dev reboots. +- **althing v2 (herald + receiver) = systemd-supervised on nh3-dev (the althing DEV box).** + Formalized this session: `althing-herald.service` (Restart=always, **Environment=PATH incl + ~/.cargo/bin** — the fix for the silent pane-dispatch outage) + `althing-receiver.service` (v2 → + pillar-3 `/owner/*` live). Stale forseti unit removed; both on v2.0.0b2, canonicalized by forseti. + `reference_nh3_dev_althing_herald`. -- **granite-4.1-8b bf16 kept** at `irv-ml1:/home/lkraven/granite-4.1-8b-bf16` (17G, - reap-on-request; the T1-harness-spike base). comfyui was borrowed off the A6000 for the - spike + RESTORED healthy. +- **glm-5.2 = z.ai 1M-in / 128K-out, NO gateway cap** (probed z.ai; recorded in the config comment + + `reference_litellm_gateway`; commit 624a07e). All GLM entries are pure z.ai passthrough — the + effective ceiling is z.ai's canonical, not a gateway limit. -- **Recently completed (2026-06-22..25, now in Recent decisions):** Worldtree #314/#322/#317 - persona-render config arc; nh3-extdev althing v0.17.1 mesh peer (Model B); zellij web-seat pilot. +- **`/books` mounted (transient) on nh3-dev** (`10.0.50.50:/mnt/books` → NFSv4 ro,soft; re-mount via + `infra-ops@10.100.10.50` if it reboots). -- **Backups — recovered + hardened (2026-06-20), STILL OPEN:** rotate the 5 disclosed - rest-server creds (operator, offline); confirm esh-vm-db's resticprofile includes DB - dumps (esh-pve-nas restic-only). Topology + 2-min check: `docs/runbooks/backups.md`. +- **Backups — recovered + hardened (2026-06-20), STILL OPEN:** rotate the 5 disclosed rest-server + creds (operator, offline); confirm esh-vm-db's resticprofile includes DB dumps. `docs/runbooks/backups.md`. -- **R22 (brokkr/dwarves) — CONCLUDED, gateway-only.** Phase B (internal-model hooks) - CANCELLED (Worldtree model-agnostic → no deploy path). Whole footprint = gateway calls - to free `qwen3.5-122-a10b` via a PERSISTENT full-access key at - `/home/lkraven/.r22-gateway-key` (mode 600, carries paid GLM, do NOT delete). Detail in - Recent decisions. +- **Open follow-ups (low-priority):** clean phantom `qwen3.6-35b-a3b` off the gateway (lists in + /v1/models but 400s); docker-daemon default log-cap on ana-docker; drop the now-inert + `mood.decay_rate`/`mood.stale_hours` keys from the deployed `/opt/worldtree*/config` bind-mounts + (harmless-present per worldtree-dev; edit-only/no-restart at a no-deploy window). -- **Open follow-ups (low-priority):** clean phantom `qwen3.6-35b-a3b` off the gateway - (lists in /v1/models but 400s — backend displaced); docker-daemon default log-cap on - ana-docker (systemic fix behind the 94 GB disk incident); drop the now-inert - `mood.decay_rate`/`mood.stale_hours` keys from the deployed `/opt/worldtree*/config` - bind-mounts (removed from canonical in R30 b15–b17; harmless-present per worldtree-dev - since `CharacterSchema.mood` is `dict[str,Any]` — clean edit-only/no-restart at a - no-deploy window to restore byte-identity; new optional `mood.tau_base` is unset→derived). +- **Standing / parked:** Mac Pro migration (hardware-gated); R17 v2 corpus HELD; disclosed-keys + rotation queue; clean legacy `news-digest`; R22 gateway-only full-access key at + `/home/lkraven/.r22-gateway-key` (mode 600, carries paid GLM, do NOT delete); granite-4.1-8b bf16 + at `irv-ml1:/home/lkraven/granite-4.1-8b-bf16` (reap-on-request); Deckard staged on ana-ml2 as + T1's writing benchmark. -- **Standing / parked (from prior):** Mac Pro migration (`migration-plan.md`, - hardware-gated); R17 v2 corpus push HELD (distribution barred); disclosed-keys rotation - queue; clean legacy `news-digest` on ana-docker. - -- **Worldtree config-propagation (reference):** demo+personal bind-mount config - (`providers.yaml`/`model_roles.yaml`/`policies.yaml`/`defaults.yaml`, now incl. the - #314/#322/#317 additions) from `/opt/worldtree{,-personal}/config` - (infra-ops-deployable, byte-identical from worldtree-dev canonical sha); pinned is - IN-IMAGE. Reload via `docker restart `, NEVER `compose up` (stale-`:latest` - footgun); WT PDP is rule-based (a tier needs allow RULES, not just scopes). **#317 - lesson: config REMOVALS are NOT backward-compatible with the still-running image — push - promptly, don't bounce the instance in the window.** +- **Worldtree config-propagation (reference):** demo+personal bind-mount config from + `/opt/worldtree{,-personal}/config` (infra-ops-deployable, byte-identical from canonical); reload + via `docker restart `, NEVER `compose up` (stale-`:latest` footgun). **The role registry + is loaded ONCE + CACHED at startup** (`ConversationService.role_registry`) → a bind-mount + `model_roles.yaml` change needs a container restart to take effect; pre-stage the bind-mount + BEFORE the activating deploy's restart (else it needs a second bounce). config REMOVALS are NOT + backward-compatible with the still-running image. ## Recent decisions -- `[2026-07-02]` **mtf-dev granite harness-spike provisioned on irv-ml1 (comfyui displaced - for the A6000) → ran GREEN.** Freed the A6000 by stopping comfyui (operator-coordinated - with comfy-dev), staged granite-4.1-8b bf16 to `irv-ml1:/home/lkraven`, mtf-dev's TRL - SFT→DPO→eval seam proved end-to-end (DPO genuinely learned, 0.833 acc; anti-slop ~0 = the - expected null on clean-writing granite-instruct); comfyui restored healthy. De-risks T1's - trainer harness ahead of the real qwopus train. **MECHANICAL green ONLY — efficacy NOT - validated, by design** (mtf-dev confirm 2026-07-02): the run used a 12-row/12-pair - SYNTHETIC writing fixture (clean-vs-sloppy generic prompts), NOT the E-RP corpus; rank 8, - 1 epoch, ~2 SFT + ~6 DPO steps. The in-loop base-vs-adapter check (HoldoutEvaluator) - already ran and returned the expected null (anti_slop_improvement −0.002); the adapter was - then reaped (gone), so there is nothing to A/B — and on a mechanics-only synthetic adapter - an A/B would only reconfirm the null. Real efficacy = the T1 run (real recipe + E-RP data - on qwopus). **Operator DECIDED 2026-07-02: DEFER efficacy to the T1 run** — NO intermediate - real-efficacy granite spike (uninterpretable proxy: granite arch ≠ qwopus MoE+GDN AND - censored-instruct vs qwopus-abliterated, so a null could be refusal priors fighting E-RP - rather than a recipe bug). If any pre-T1 de-risk is later wanted, the right target is the - real-data/judge WIRING check (corpus tokenizes → sane DPO pairs → coherent judge scores; - arch-independent, validates LitBench-less) — NOT a granite train. mtf-dev holds; infra's - next owed item stays the queued swappable-LoRA-on-NVFP4 load test (gated on first T1 adapter). +- `[2026-07-05]` **T1 training venue: CLOUD recommended; operator chose smoke-first on ana-ml2.** + On-prem ruled out (ana-ml2 full — both 96G GPUs ~93G used): keep-serving = NVMe offload ~6-8 DAYS; + full ana-ml2 shutdown = CPU offload ~1 DAY but a whole-fleet outage. Cloud Vast.ai 8×80GB (no + offload → ~3-6h, ~$60-500, zero fleet impact) is the clean alt (mtf-dev + infra-ops both rec; + Vast for its no-content-AUP marketplace + likely-existing VastBlue account). Operator's next step + = the ana-ml2 CPU-offload SMOKE (~60 steps) to get real samples/sec before the full-outage-vs-cloud + call. HF base verified public (zero-auth pull). Runbook + gotchas in `reference_t1_cloud_train_plan`. -- `[2026-07-01]` **Worldtree #332 embed-recall diagnosed + scoped-log view/tunnel - provisioned + fix verified.** The persona-recitation gate re-embedded persona segments - every turn (15× re-embed, 95% cross-turn recurrence, one anchor ×206/min → worldtree-gateway - was ~95% of the embedding backend load); worldtree-dev's cross-turn content-hash cache - dropped it to 1.01× in prod. Built a boundary-safe read-only `wt_gateway_logs` view + - `wt_readonly` role + `wt-db-tunnel` (corviduo-dev) for their regression-watch. auto-memory - `reference_wt_gateway_scoped_log_view`. +- `[2026-07-05]` **glm-5.2 canonical limits recorded** (probed live vs z.ai): **1,048,576 (1M) input + context / 131,072 (128K) max output**; NO gateway-side cap (pure passthrough → z.ai's limits are + effective). Written to the config comment (commit `624a07e`) + `reference_litellm_gateway`. -- `[2026-07-01]` **qwopus native MTP speculative-decode tested on `gen` → NOT kept.** qwopus - HAS a full native MTP head (785 tensors, in NVFP4 + bf16). Measured +12% single-stream but - **−15–20% AGGREGATE at moderate concurrency** (N=4: 250→200 tok/s) + it silently drops - `min_p`/`logit_bias` → reverted to clean baseline. The win is banked for T1 (MTP as a - per-deployment option, preserved through requant). `reference_gen_qwopus_122b`. +- `[2026-07-04]` **character-rp: gateway-forwarding proven + role pre-staged + #344 shipped.** + Empirically confirmed per-request `extra_body` (top_k/repetition_penalty) forwards through the + `gen-reasoning` LiteLLM alias to vLLM + standard params override the alias defaults — no gateway + cap needed (I over-built a dedicated alias, operator corrected, reverted with zero fleet impact). + Pre-staged the `character-rp` role into demo+personal bind-mount `model_roles.yaml` (byte-verified + on b18; caught the cached-registry ordering). worldtree-dev shipped **#344 (v1.0.0b19)** for the + durable-agent override-drop. spend_logs spot-check queued (ratatoskr's timestamp ping). -- `[2026-07-01]` **Deckard trial → reverted to qwopus (`gen`).** `robbatt/Qwen3.6-40B-Deckard-NVFP4` - won writing "in every way" but at ~36 vs ~90 tok/s (dense-40B vs MoE-~10B-active); - spec-decode rescue ruled out (aeon DFlash image is arm64/DGX-Spark-only; Deckard's - NVFP4/bf16/base-27B all LACK MTP; EAGLE-head training a multi-week non-starter). git - `b63c48b` (repoint) + `681eb70` (revert). Deckard kept staged as T1's writing benchmark. +- `[2026-07-04]` **althing v2 herald+receiver formalized as systemd on nh3-dev.** `althing-herald.service` + (Restart=always, **Environment=PATH incl ~/.cargo/bin** — the pane-dispatch fix) + `althing-receiver.service` + (v2 → pillar-3 `/owner/*` live); stale forseti unit removed; both on v2.0.0b2, canonicalized by + forseti. `reference_nh3_dev_althing_herald`. -- `[2026-06-25]` **althing re-architected to the v0.17 lean multi-machine bus; nh3-extdev - stood up as a mesh peer (MODEL B).** v0.15.0's lean-bus cut RIPPED - moderation/chamber/forseti-daemon/agent-runner/redis-valkey; **v0.17 = per-box - local-SQLite bus + a courier/receiver for P2P over the 10.x net** (installed per-box via - `uv tool install`, NOT CI). Migrated nh3-dev's client v0.15→v0.17. nh3-extdev - provisioned as a full mesh peer with the operator's **MODEL B** (dedicated `althing-svc` - service account + group-shared `/srv/althing` root, so multiple OS users share one - config/DB; `ALTHING_ROOT` via /etc/profile.d). Verified multi-user concurrent rw. - Supersedes the old althing Tools-row (was chamber/daemons/valkey). auto-memory - `reference_nh3_extdev_althing_mesh`. +- `[2026-07-04]` **LitBench-RM respun (irv-ml1 A6000, comfyui displaced)** for T1's reward ensemble; + operator sole comfyui consumer, holding image-gen until LitBench done. `reference_litbench_rm_irv_ml1`. -- `[2026-06-23]` **zellij native web client piloted on nh3-dev** (`zellij-web.service` - :8443, native TLS), system-installed for all users, ALONGSIDE ttyd (kept as the no-auth - fallback). Bind left `0.0.0.0` per operator. auto-memory `reference_zellij_web_seat`. +- `[2026-07-03]` **ratatoskr-dev DEMO Heimdall key provisioned (R30 φ0).** Minted a tier-user key on + the demo via `POST /admin/keys` (bootstrap admin key), mirroring their personal base consumer (no + character-binding); base-agent affect reads work ungated. `reference_worldtree_demo_key_mint`. -- `[2026-06-22]` **Worldtree persona-render config arc (#314 relational-stance / #322 - affect-mood gate / #317 valence teardown) pre-synced + deployed green on demo+personal** - — #317 a boot-blocking config REMOVAL (relational_valence). Config-sync recipe + - additions-vs-removals lesson in auto-memory `reference_corviduo_dev_emergency_ops`. +- `[2026-07-02]` **mtf-dev granite harness-spike ran GREEN — MECHANICAL only, efficacy DEFERRED to + the T1 run.** Trainer TRL SFT→DPO→eval seam proven end-to-end on a synthetic fixture (not the E-RP + corpus); operator DECIDED no intermediate real-efficacy granite spike (uninterpretable proxy — + arch gap + abliteration axis). `reference_gen_qwopus_122b`. -- `[2026-06-20]` **R22 (brokkr/eitri/dwarves) stood down to gateway; full-access R22 - key minted; Phase B parked.** Operator caught the raw-weights/GPU ask as premature — - Phase A (P00 + NL-rendering) is PROMPT-LEVEL on the LiteLLM gateway, not raw weights - (no GPU, no Blackwell displacement). Real blocker was gateway ACCESS: the shared - `all-agents-local` key only serves granite chat, and granite-8b is too weak to be the - P00 model-under-test (capability-confound). Operator said "full access" → minted - `r22-brokkr-phaseA` LiteLLM key (scope **all-proxy-models**, incl PAID GLM), dropped - `~/.r22-gateway-key` (mode 600) for eitri — **re-minted PERSISTENT** after eitri's - stateless tool-env couldn't hold it in env + shredded the one-shot drop (old key - revoked); the live full-access key now lives at `/home/lkraven/.r22-gateway-key`, - read per-command, **don't delete** (cost surface — carries paid GLM; operator - reconfirmed full-open + declined to narrow, 2026-06-20 — settled, don't re-surface). - Lesson: stateless consumers need a persistent mode-600 file read per-call, NOT - shred-after-load. **P00 MUT = `qwen3.5-122-a10b` (alias - `gen`)** — 122B, capable, FREE-LOCAL (no per-token cost); told them keep frozen - calibration on it, NOT the paid GLM the key can now reach. **PHASE B** (raw-weights - hooks: Dvalin activation-steering on hidden states + Regin ECM logit-injection on a - 7-8B Qwen/Llama via transformers LogitsProcessor) **CANCELLED 2026-06-20 (not - deferred)** — operator cut the internals arms: **Worldtree is model-agnostic** (regard/ - affect renders via context manipulation = prompt-level only), so the internal-model - approach has no deploy path (the pragmatic-outcome gate firing). R22's ENTIRE compute - is now the gateway (free qwen3.5-122-a10b MUT); NO raw-weights/GPU ever; the A6000 - slice reservation is released (was never provisioned). The full-scope R22 key stays - (pinned to the free qwen-122, GLM untouched). **OPERATOR STEER (2026-06-20): R22 is - research for a PRAGMATIC/deployable outcome, NOT advancing-the-art.** Before the dwarves - invest in Phase B, establish a deploy path (distilled steering-vector / fine-tune - applied to a model we already serve). Operator is taking the dwarf discussion directly. +- `[2026-07-01]` **Worldtree #332 embed-recall diagnosed + scoped-log view/tunnel provisioned + fix + verified** (15×→1.01× re-embed). `reference_wt_gateway_scoped_log_view`. -- `[2026-06-20]` **claude-bot issue-scope token minted for worldtree-dev self-serve** - (closes their last `tea`-as-vh fallback — issues; they already self-serve Actions + - deploys). Gitea PAT scopes are IMMUTABLE → minted a NEW claude-bot PAT - `worldtree-dev-actions-issues-20260621` (id 16) with `write:repository`+`write:issue` - via basic-auth (`claude-bot:gitea-password`, POST `/users/claude-bot/tokens`); dropped - mode-600 `~/.claude-bot-token-issues` → worldtree-dev swaps. **Gitea COLLAPSES - `read:issue` into `write:issue`** (write implies read). Old token (id 15) REVOKED after - verification → claude-bot carries ONE consolidated worldtree-dev token (id 16) + arbo-ci - (id 14). Advances the credential-migration directive. +- `[2026-07-01]` **qwopus native MTP speculative-decode tested on `gen` → NOT kept** (+12% single-stream, + −15–20% aggregate at concurrency, silently drops min_p/logit_bias). Banked for T1. `reference_gen_qwopus_122b`. -- `[2026-06-20]` **rest-server-ana recovered + backup prevention shipped + worldtree-dev - admin keys provisioned** (operator-directed). rest-server-ana fixed (mount-a + - force-recreate via `infra-ops@ana-docker`); freshness alert + fstab hardening landed - (see `docs/runbooks/backups.md`). Minted worldtree-dev **admin-tier Heimdall keys** on - demo (key_id d113207c) + personal (f4f75adb), dropped mode-600 to - `~/.wt-admin-{demo,personal}` on nh3-dev → worldtree-dev now self-serves key minting. - Cred rotation (5 rest-server pw) BELAYED per operator. +- `[2026-07-01]` **Deckard trial → reverted to qwopus (`gen`)** (won writing "in every way" but ~36 vs + ~90 tok/s; spec-decode rescue ruled out). git `b63c48b`→`681eb70`. Deckard kept staged as T1's + writing benchmark. -- `[2026-06-20]` **claude-bot → ADMIN on vh/Worldtree** (operator-authorized; - one-time use of operator vh-admin to enable the migration) — claude-bot - self-serves WT deploys/tokens henceforth, reducing personal-cred fallbacks. +- `[2026-06-25]` **althing re-architected to the lean multi-machine bus; nh3-extdev stood up as a + MODEL B mesh peer** (dedicated `althing-svc` + group-shared `/srv/althing`). `reference_nh3_extdev_althing_mesh`. -- `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.** Stand up service accounts (claude-bot Gitea user + scoped tokens, distinct litellm admin key); re-mint consumer creds under them; flag personal-cred fallbacks until done. (auto-memory `project_migrate_infra_access_to_claude_credentials`) +- `[2026-06-23]` **zellij native web client piloted on nh3-dev** (`zellij-web.service` :8443) + alongside ttyd. `reference_zellij_web_seat`. + +- `[2026-06-22]` **Worldtree persona-render config arc (#314/#322/#317) pre-synced + deployed green + on demo+personal** — #317 a boot-blocking config REMOVAL. `reference_corviduo_dev_emergency_ops`. + +- `[2026-06-20]` **R22 (brokkr/dwarves) stood down to gateway-only; full-access R22 key minted; + Phase B CANCELLED** (Worldtree model-agnostic → no deploy path). Key at `/home/lkraven/.r22-gateway-key` + (persistent mode-600, carries paid GLM, don't delete). MUT = free `qwen3.5-122-a10b` (`gen`). + Operator steer: R22 research is gated on a pragmatic/deployable outcome, not advancing-the-art. + +- `[2026-06-20]` **claude-bot issue-scope token minted for worldtree-dev self-serve** (id 16, + `write:repository`+`write:issue`); old token revoked. Advances the credential-migration directive. + +- `[2026-06-20]` **rest-server-ana recovered + backup prevention shipped + worldtree-dev admin keys + provisioned** (demo d113207c / personal f4f75adb). Cred rotation (5 rest-server pw) BELAYED. + +- `[2026-06-20]` **claude-bot → ADMIN on vh/Worldtree** (operator-authorized) — self-serves WT + deploys/tokens henceforth. + +- `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.** + (auto-memory `project_migrate_infra_access_to_claude_credentials`) _118 older entries archived to archival-memory.md._ ## Tried and abandoned -- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / - unauthorized" is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` - is PULL-ONLY ("image must already be built by a push to main") but fires on the - `staging/vX` tag push SIMULTANEOUSLY with `deploy.yml`'s main build → it tries to pull the - image ~2.5 min BEFORE the build finishes pushing it → step-4 "Verify image exists" aborts - "not found". Misattributed to registry-auth twice (the earlier cff3328 saga too). DIAGNOSE: - the image tag (12-char short-sha, NOT 7) exists in the registry + the VM login succeeds ⇒ - it's the race. FIX: re-run once the build's done (image now present), OR gate deploy-personal - on `workflow_run: completed`. +- `[2026-07-04]` **LiteLLM (this gateway version) mutates the SHARED deployment config in-place on + per-request sampler-param merge** → my deliberately-invalid `top_k=-5` forwarding-probe bled into a + param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT + caching (none configured), NOT a config change. **Never fire invalid/distinctive sampler values at + a SHARED gateway alias with live consumers** — use a throwaway alias, or a `docker restart litellm` + flushes residual carryover. `feedback_litellm_shared_param_mutation`. + +- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools + needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The + althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO + routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`. + +- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe + ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast + on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the + 566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`. + +- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" + is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but + fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push + finishes. FIX: re-run once built, or gate on `workflow_run: completed`. - `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS - moderate-concurrency aggregate throughput + silently ignores `min_p`/`logit_bias`.** - Measured on qwopus `gen`: N=1 +12%, N=4 −20% aggregate. Don't bolt spec-decode onto the - fleet `gen` for a single-stream win — reserve it for dedicated/interactive deployments. + moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%, + N=4 −20%). Reserve for dedicated/interactive deployments. -- `[2026-07-02]` **irv-ml1 `/worktank` ROOT is root-owned — lkraven can't write there (and - irv-ml1 sudo needs a password non-interactively) → stage model pulls to `/home`.** Also - PIN THE A6000 BY UUID for training runs: nvidia-smi index 1 is the A6000, but native-CUDA - ordering can differ vs docker, and the 3090 (index 0) is usually near-full → land there and - OOM. `CUDA_VISIBLE_DEVICES=GPU-`. +- `[2026-07-02]` **irv-ml1 `/worktank` ROOT is root-owned — lkraven can't write there (irv-ml1 sudo + needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA + ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-`. -- `[2026-06-25]` **althing "unreachable: — retry later" can MASK an app-level - 500.** A multi-hour "intermittent :8087 / firewall flap" hunt was a red herring: raw - network was always clean (curl POST to the receiver `:8087` worked; a connect probe = 0 - fails). Root cause = the receiver's DB agents-table wasn't synced with the config roster, - so delivery 500'd `"unknown to: "`, which the courier MAPPED to "unreachable" - (looks like network/DNS). **Diagnose: raw curl to :8087 + a connect-probe pass ⇒ NOT - network — look up-stack.** Fixed in althing **v0.17.1** (receiver auto-ensures the - recipient on delivery). Stop-gap on older builds: `ALTHING_HANDLE= althing-cli inbox` - as the receiver user syncs config→DB; re-run after any roster change. auto-memory - `reference_nh3_extdev_althing_mesh`. +- `[2026-06-25]` **althing "unreachable: " can MASK an app-level 500.** Raw network was + clean; root cause = receiver DB agents-table not synced with the config roster → delivery 500'd + "unknown to: ", MAPPED to "unreachable". Diagnose: raw curl to :8087 + connect-probe ⇒ NOT + network. Fixed in althing v0.17.1. `reference_nh3_extdev_althing_mesh`. -- `[2026-06-20]` **rest-server `.htpasswd: permission denied` = the ana-nas - NFS mount FAILED (ghost file on the local mount point), NOT a decommission.** - `mnt-backup.mount` stuck `failed` (fstab bare `defaults`, no retry) → the - rest-server serves an empty local dir with a root:root 0-byte `.htpasswd`. - Documented recovery in disaster-recovery.md. Don't tear down what looks like - a crash-looping legacy container until you've checked fstab + the docs — it - was the live ana-side restic target. +- `[2026-06-20]` **rest-server `.htpasswd: permission denied` = the ana-nas NFS mount FAILED (ghost + file on the local mount point), NOT a decommission.** `mnt-backup.mount` stuck `failed` (fstab bare + `defaults`) → rest-server serves an empty local dir. Recovery in disaster-recovery.md. -- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — - but `ssh infra-ops@ana-docker` HAS NOPASSWD root** (corrected later same day). - Early on a `sudo cp` as lkraven silently failed (password prompt) → one - wasted gateway bounce; I then wrongly concluded "no sudo on ana-docker" and - nearly punted the rest-server-ana recovery to the operator. The real rule: - reach for `infra-ops@ana-docker` for sudo ops; lkraven-owned files (litellm - config, most stack compose/conf) take plain `cp` under either identity. +- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — but `ssh + infra-ops@ana-docker` HAS NOPASSWD root.** A `sudo cp` as lkraven silently failed → nearly punted + the rest-server recovery. Reach for `infra-ops@ana-docker` for sudo ops. _98 older entries archived to archival-memory.md._