vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d1bea1399406b58b9efcb3bbf571c0c9e92a1ede
esh-pfi-infrastructure/persistent-memory.md
T

45 KiB
Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-06-16

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo Role CI status
vh/task-board MCP + web dashboard for assistant task state (port 7878) push-to-main → CI deploys (2026-04-29)
vh/vor Inquisitor UI sidecar (port 7879) push-to-main → CI deploys (2026-04-29)
vh/nevermore Twice-daily LLM-curated briefing (port 8181, replaces news-digest) push-to-main → CI deploys (2026-04-30)
vh/asset-engine Internal control plane over inference services (port 8200, LAN-direct) push-to-main → CI deploys (2026-05-12)
vh/althing Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) push-to-main → CI deploys (2026-05-14)
vh/mead-hall Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) push-to-main → CI deploys (2026-05-16)
vh/skaldsong Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) push-to-main → CI deploys (2026-05-19)
vh/worldtree Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration push-to-main → CI deploys
vh/yt-voice-clipper YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see docs/runbooks/ytvc-autodeploy.md
vh/arbo Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) push-to-main → gitea-webhook auto-deploy to irv-ml1 (catalog-only restart; infra side = stacks/arbo/)

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

  • Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix.

  • pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

  • Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. For personal-instance admin ops, fetch the bootstrap admin per-op via docker exec worldtree-personal-worldtree-api-1 printenv WORLDTREE_BOOTSTRAP_ADMIN_KEY on corviduo-dev. Used for POST /admin/keys, admin diagnostics (/admin/sessions/<id>/{bifrost,tools}, etc.).

  • Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637 (nh3-dev iteration), skaldsong:7c1dbbbe (ana-docker prod), althing:50d85460, mead-hall:a360822d. Same user_id=skaldsong across both skaldsong keys → shared Heimdall agent slot; different key_id → independently rotatable. Pattern: mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

  • Skaldsong CD pattern (registry-pull). Differs from althing / asset-engine which build-on-host. vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only (no :latest health-gated advance yet). Prereq: host needs docker login gitea.phasefinal.com once (read:package PAT) — not currently in the workflow.

  • gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44, ana-srv1) — the public path fail2bans the host egress IP and wedges webhook deploys. :22 on 10.250.50.70 is ana-docker's HOST sshd, not gitea. Full gotcha in docs/orientation.md → Git/gitea.

  • docker-as-root pattern (for ops that have no admin API, e.g. SqliteUserStore.set_bifrost_credentials): on hosts where the SSH user is in the docker group but lacks passwordless sudo, run docker run --rm -v <target-dir>:/wt -v /var/run/docker.sock:/var/run/docker.sock docker:cli sh -c "..." to edit deploy-owned files without sudo. Documented with security warning in servers/corviduo-dev/README.md. docker-group membership is effectively root via bind-mount; treat as a sudo-equivalent grant. Foot-gun: when running docker compose inside this sandbox, any relative path in compose.yaml (e.g. ${WORLDTREE_CONFIG_DIR:-./config}) resolves against the sandbox CWD, but Docker daemon interprets the resulting path against the HOST filesystem. Always pass -e VAR=/abs/path to the docker run invocation for any relative-default config dir.

  • scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step. That prompt is interactive → elway can't run unattended from a non-TTY tool if any step needs sudo. For sudo-free playbooks (no sudo: true steps) it runs fully non-interactive over key SSH. To create root-owned dirs WITHOUT host sudo, use the docker-daemon-root trick: docker run --rm -v /worktank:/mnt alpine sh -c 'mkdir -p /mnt/<x> && chown -R 1000:1000 /mnt/<x>'.

Current state / in-flight

As of 2026-06-16:

  • ratatoskr Tier-3 MEMORY plane wired + live-smoke driven — persist+dispatch GREEN, recall-injection OPEN (handed off, OUT of loop per operator). The missing server-side step was the per-PORT url-guard allowlist: added 10.100.10.50:8391 to personal's BIFROST_CLIENT_ALLOWED_HOSTS (affect :8390 was already in; via playbooks/wire-personal-worldtree-memory-allowlist.yaml, c99aa49). Consumer key REUSED (plane-agnostic, no mint). Drove define→bind→persist(async upsert_many 200)→recall(search 200); ratatoskr's memory.db durable + 1024-dim indexed. Promotion gate = a consumer agent defined WITH a memory block (ctx.memory_config≠None), NOT agent_self_enabled. Cross-session recall-INJECTION root-caused to a SCORE-SHAPE seam: Worldtree's injector reads score_breakdown.{similarity,salience}, but ratatoskr's search() returns top-level score + chunk.salience (0.9, ≥0.85) → both gate-reads collapse to 0 → filtered AFTER a clean 200. Handed to worldtree-dev (#295 recall leg OPEN; their ADR-0009 wire-spec call — bifrost returns score_breakdown, or Worldtree reads existing fields). (auto-memory reference_bifrost_plane_wiring)

  • Worldtree demo + personal character model = mistral-small-4 (flipped from qwen3.6-35-a3b, 2026-06-16) — reordered model_roles.yaml character.binds mistral-first (first bind = default), qwen kept in the switch-allowlist; applied via pin-safe recreate, fresh-agent resolution verified. Backups model_roles.yaml.bak-pre-mistral-character.

  • ana-ml2 GPU layout RESHAPED again (2026-06-15/16) — both cards now full with NVFP4 tenants. GPU 0 = Mistral Small 4 (mistral-small-4 stack, mistralai/Mistral-Small-4-119B-2603-NVFP4, 119B/6.5B-active MoE, :8010, gateway mistral-small-4 + mistral-small-4-reasoning@effort=high). Pinned vLLM v0.22.0 — the LAST release with working Mistral vision (#44911 fetch_images regression breaks it on 0.22.1+/0.23.0). Serves the full native 256K context (max-model-len 262144, max-num-seqs 32 — fits the tight card, ~5 GB free). Text + vision both work; reasoning via reasoning_effort (BINARY: none|high). Dedicated single-tenant; the operator's creative-writing model (abliteration planned → it succeeds llama-swap). GPU 1 = qwen36 swapped FP8→NVFP4 (nvidia/Qwen3.6-35B-A3B-NVFP4, fp16 KV, util 0.34, :8007, gateway name qwen3.6-35b-a3b UNCHANGED + -thinking variant) — the ModelOpt NVFP4 MoE LOADS on 0.23.0 now (the 2026-06-14 "blocked" finding is RESOLVED). + granite restored (0.34/131072) + Selene FP8 judge added (selene-1-mini-8b, AtlaAI Selene-1-Mini-Llama-3.1-8B dynamic fp8, util 0.17, :8011, ctx 32768) + embed/rerank/reward. ~5.6 GB free. Prefix-caching ON on all 4 generative. llama-swap is DOWN (decommissioned from GPU 0 for Mistral; its qwen GGUF consumers migrated to the gateway). (auto-memory reference_nvfp4_moe_loads_on_vllm_023)

  • arbo engine builds handed to comfy-dev; Gitea Actions runner LIVE on irv-ml1. Operator approved comfy-dev owning arbo engine deploys (deploy-engine.sh, build-local). Runner = irv-ml1-arbo (act_runner v0.2.13, host-executor as lkraven, repo-scoped to vh/arbo, labels self-hosted+irv-ml1, systemd unit act_runner.service, polls the INTERNAL gitea route). comfy-dev to commit .gitea/workflows/deploy.yml — Phase 1 = engine-only auto-deploy (catalog stays on the :9009 webhook), Phase 2 catalog cutover later. infra-ops = box-level escalation only (runner, the package:write PAT once corviduo-org exists, webhook, the arbo-deploy.sh HEAD-baseline restart-detection bug still open).

  • claude-bot Gitea service account STOOD UP (credential-migration foundation) — user id=3, non-admin/scoped, token arbo-ci=write:package+read:repository, creds at ~/.config/claude-bot/{gitea-password,gitea-token} (mode 600). Registry-push DEFERRED: vh is a USER not an org, so Gitea won't grant a scoped service account package-write to vh/*. (auto-memory project_migrate_infra_access_to_claude_credentials)

  • Worldtree demo + personal MIGRATED to v0.35.10 capability-named profiles — persona substrate moved qwen→Mistral (profiles dialogue/classification→ mistral-small-4, deep-reasoning→mistral-small-4-reasoning), Domari→selene; qwen heretic/heavy retired. Both green. The win: future model swaps (the abliterated Mistral) are now bind-mount-only providers.yaml edits — no image/agent/coupling.

  • R18 clip+caption auto-assist models STAGED on ana-ml2 (brokkr/Eitri) — WhisperX + PANNs CNN14_DecisionLevelAtt + Qwen3-Omni-30B-A3B-Captioner at Eitri's /srv/r18-clip-caption path contract (symlink → /tank; dedicated r18clip user, workspaces/ 0700). Stub-level offline smoke PASSED (lo-only netns); Eitri closed the scaffold deliverable. AWAITING: Eitri/brokkr's concrete model runners (replace the deterministic stubs) → re-smoke the real path; AND the hard real-voice gate — no real victim voice through the shared-box captioner until ephemeral-(b) controls (in-VRAM-only, no shared-disk persistence, access-logged + auditd stood up) are built AND explicitly confirmed to brokkr. (auto-memory project-r18-clip-caption-staging)

  • MastMed prototype LIVE + public — gated static-SPA container on ana-docker:8089 (caddy basic_auth, user mastmed) fronted by a cloudflared quick tunnel (ephemeral *.trycloudflare.com — dies on tunnel-container restart → new URL). Swap to a named tunnel (operator's scoped CF token + hostname) for a stable URL anytime; container + auth unchanged.

  • comfyui (irv-ml1) VAE-decode SEGFAULT — RESOLVED 2026-06-15. Root cause was comfy_aimdo's cudaMalloc hooks conflicting with ComfyUI's load-bearing --disable-cuda-malloc: with the optimized backend disabled (cu129<cu130) but the hooks still installed, VAE-decode allocation hit the dead path → native segfault. NOT cu130 (that would re-enable the allocator we deliberately disable). comfy-dev self-fixed: patched comfy_aimdo/control.py init() → return False (aimdo was non-functional anyway), folded into reprovision-venv.sh (their commit d5a2233). Verified clean across zimage/flux/SDXL.

  • Arbo prod: engine at v0.11.38 (three engine-only fixes 2026-06-16 for granite IR-shape "AI unavailable" degradations: intent_summary default, bare-string→list coercion); now comfy-dev-owned (see the runner entry above). Registry still holds NO arbo image (local-build-only) — that gap closes when the corviduo-org + package:write token land (deferred). Auth OFF (WireGuard boundary).

  • R16 splice-pivot yield probe RAN 2026-06-14 (was "awaiting go"). Inline-gen arc stays CLOSED (v1 @ default decode); the splice pivot's cheap de-risk is now executed: 45 standalone NVV ([moan_soft]/[moan_intense]/[groan] × seeds 1-15) from Eleanor.wav on the v4 adapter + objective metrics (Resemblyzer spk-cosine, MFCC-dist, F0, parselmouth HNR) → /mnt/smithy/scratch/r16_audition_yield, served on brokkr-audition :8137. 45/45, 0 degenerate; spk-cosine 0.48-0.81. Brokkr notified. NEXT: operator ear-bin (usable/impure) → brokkr's yield% / survivorship / identity analysis. gen_yield_probe.py in irv-ml1:~/r16-vmoan-harness.

  • R17 v2 corpus characterization still on the local irv-ml1 branch, push HELD (r17-v2-characterization). No-push rule AND ASR-content-exposure call on the intimate-audio batch. Awaiting brokkr-collect or explicit push approval. (LOCAL-ONLY soundgasm-derived corpus — distribution barred, never echo transcripts to the bus.)

  • Mac Pro migration is still the big open project — migration-plan.md (repo root): WORKSTATION-ONLY move of nh3-dev's Hat-1 dev env to an M2 Ultra Mac Pro on the 10.100 subnet. Hat-2 fleet sidecars STAY on the Linux VM. Phase 0 (push pushables + confirm the ~6 local-only repos) is the only time-sensitive step. Cutover = rsync working trees, NOT re-clone. Everything else waits on hardware.

  • sglang-vs-vLLM bench stack staged but parked (5f049cb) — the originating SGLang-RadixAttention question resolved (vLLM v1 defaults prefix-caching ON, now pinned explicit on granite+qwen). NVFP4 chase abandoned. MAX_JOBS-on-shared-prod foot-gun caps any from-source build on ana-ml2.

  • pi + GLM 5.1 harness live on nh3-dev — glm launcher runs pi against glm-5.1 (thinking-off) via the LiteLLM gateway; glm-5.1-reasoning for opt-in thinking.

  • Worldtree config-propagation lane is mature + humming — pre-merge delta-ping → sync-on-merge to demo+personal bind-mounts; pinned stays pre-cutover.

  • Disclosed-keys hygiene queue (rotate at convenience): HF token, the wt-personal keys, Gitea runner reg token, MINIFLUX_PASSWORD, ZAI_API_KEY. (The shared all-agents LiteLLM key is intentional, not hygiene-debt.)

  • Still open from prior: clean legacy news-digest on ana-docker; the docker push 60s ceiling mystery uninstrumented.

Recent decisions

  • [2026-06-16] ratatoskr Tier-3 MEMORY plane wired + live-smoke driven. The missing server-side step was the per-PORT BIFROST_CLIENT_ALLOWED_HOSTS allowlist (:8391; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the score_breakdown shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a memory block (ctx.memory_config≠None), not agent_self_enabled. (tracking: c99aa49 + auto-memory reference_bifrost_plane_wiring)

  • [2026-06-16] infra-ops = the DURABLE admin path on corviduo-dev (operator granted standing admin; extends the 2026-06-15 bootstrap) — added a Host corviduo-dev → User infra-ops ~/.ssh/config alias + repointed servers/corviduo-dev/ssh-target to infra-ops@; ssh corviduo-dev / elway corviduo-dev now land as infra-ops w/ NOPASSWD (vh = explicit-only human path). elway resolves via ssh-config, NOT the ssh-target file. (tracking: c99aa49 + auto-memory reference_corviduo_dev_emergency_ops)

  • [2026-06-16] demo + personal character model flipped qwen3.6-35-a3b → mistral-small-4 (operator) — in model_roles.yaml the FIRST catalog_id in a role's binds: is the default; reordered mistral-first (qwen retained in the switch-allowlist, non-destructive), applied via the PIN-SAFE api recreate (derive WORLDTREE_IMAGE from the matrix sibling — never a bare compose up, which falls to the crash-blocking stale :latest), verified a fresh character agent resolves model=mistral-small-4. (auto-memory reference_corviduo_dev_emergency_ops)

  • [2026-06-16] NVFP4 ModelOpt MoE now LOADS on vLLM 0.23.0 — SUPERSEDES the 2026-06-14 "blocked" finding (#44081 fixed). Confirmed the cached nvidia/Qwen3.6-35B-A3B-NVFP4 loads clean + generates on 0.23.0; swapped prod qwen36 FP8→NVFP4 (~13 GB reclaimed on GPU 1, fp16 KV). Validated by brokkr (speed: faster single-stream, ≥parity batched) + comfy-dev (vision anatomy-judge A/B: PASS). One W4A4 tell: minor multi-step chained-reasoning slip — doesn't bite the vision judge. Mistral's NVFP4 is a SEPARATE compressed-tensors path (day-0), distinct from the ModelOpt one. (auto-memory reference_nvfp4_moe_loads_on_vllm_023)

  • [2026-06-16] Mistral Small 4 = the GPU-0 tenant (operator) — fills the slot the deferred creative-writing pick held, and supersedes llama-swap (now down). NVFP4 119B MoE, pinned vLLM v0.22.0 for working vision (0.23.0 breaks it), 256K context. Operator's creative-writing model; abliteration planned → then it becomes the uncensored-creative model the persona substrate needs. Reasoning via reasoning_effort (BINARY none|high — no medium). (tracking: c77a9aa/9a49963)

  • [2026-06-16] Selene judge restored on vLLM FP8, NOT NVFP4 — W4A4 is too aggressive for a precision judge validated at Q6_K; FP8 ≥ Q6_K fidelity, and Selene is text-only Llama-3.1 so dynamic --quantization fp8 has no vision tower to noise. selene-1-mini-8b on GPU 1 :8011. Worldtree's Domari role repointed to it. (c985ede)

  • [2026-06-16] GPU-1 buffer put to work: selene 0.13→0.17 + qwen36 0.32→0.34 — the two KV-bound generative models that gained Worldtree consumers; granite + the pooling models under-use their util, so growing THEM is wasted. (e124a2f)

  • [2026-06-16] claude-bot Gitea service account stood up; corviduo-ORG conversion TABLED (operator). claude-bot is the credential-migration foundation. The proper fix for scoped agent package-write is converting vh (a USER) → a corviduo ORG — blast radius assessed (org creation = free; arbo-first migration = cheap, no registry images; full fleet = incremental, ~8 active vh images / 50 repos) — but tabled for later. Registry-push deferred until then. (auto-memory project_migrate_infra_access_to_claude_credentials)

  • [2026-06-16] arbo engine builds → comfy-dev OWNERSHIP (operator approved) + Gitea Actions runner stood up on irv-ml1 (host-executor as lkraven). infra-ops keeps box-level escalation (runner, package:write PAT pending corviduo-org, webhook). See the runner entry in Current state. (tracking: this snapshot + arbo thread 01KV70AB2B4B…)

  • [2026-06-16] Worldtree persona substrate migrated qwen→Mistral via capability- named profiles (v0.35.8→v0.35.10; demo + personal). PRE-SYNC-FIRST is now the standing pattern for any providers.yaml/profile change: sync config to the bind-mount BEFORE the deploy image-swap restart, so new agents + new config land together (the v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread 01KV6X0M…)

  • [2026-06-15] qwen36 thinking-split at the gateway (operator) — base qwen3.6-35b-a3b = thinking-OFF default (the vision judge needs it), …-a3b-thinking = opt-in reasoning. One hybrid checkpoint + --reasoning-parser qwen3; mirrors the glm-5.1 split. The judge's enable_thinking=false is what makes the NVFP4 swap transparent. (6de0844)

  • [2026-06-15] arbo engine rebuilt v0.11.6 → v0.11.22 from main; catalog re-applied (v0.11.23) — curated /workflows footer live. src/ schema changes (Model/Lora.summary, Workflow.blurb/keywords + the /workflows endpoint) need a MANUAL image rebuild (the webhook warns-only on src//Dockerfile). Built from /worktank/arbo/repo, bumped ARBO_IMAGE in the host .env, recreated + verified healthy (auto-rollback armed, unneeded). comfy-dev's v0.11.20 catalog push had first crash-looped prod (old baked schema extra=forbid vs new git-mounted catalog fields → ValidationError); they reverted the catalog, so rebuild-from-main loaded clean, then they re-applied (v0.11.23). comfyui is a SEPARATE compose project — the rebuild didn't touch it.

  • [2026-06-15] althing-core refreshed to v0.14.1 box-wide on nh3-dev (forseti + heid both routed it; central-sequenced per DevOps-delegation since uv tool install --force swaps the shared binary all handles use). Fixes the monitor non-truncating-lock bug so stop-monitor/--stop reliably recovers orphans. The editable install (uv tool install --force --editable .) covers every nh3-dev handle; --version trued up 0.14.0→0.14.1; verified --stop works against a real orphan.

  • [2026-06-15] ratatoskr Tier-3 affect live smoke GREEN (consumer-confirmed). Brokered/minted consumer_id="ratatoskr" Heimdall HS256 key on personal Worldtree (the key IS BOTH the create-session Bearer AND ratatoskr's JWT verifier secret — plain utf-8 string, NOT base64/raw); injected ENV-ONLY to ~/.config/ratatoskr/provider.env (mode 600) on nh3-dev; added 10.100.10.50:8390 to personal's BIFROST_CLIENT_ALLOWED_HOSTS; drove define→session (handshake 200)→turn→affect.emit (200, snapshot persisted). Worldtree signs Bifrost with the caller's raw Bearer verbatim. Drive commands + claims contract in althing thread 01KV4ZYSE3….

  • [2026-06-15] Dense Qwen3-VL-32B-NVFP4 A/B'd as arbo's anatomy hero-judge vs the qwen3.6-35b-a3b MoE → dense LOST, pilot torn down, GPU 0 reclaimed. Dense NVFP4 loads clean on vLLM where the MoE NVFP4 doesn't (no expert-scale keys); ~44 tok/s on Blackwell GPU 0; vision tower stays high-precision. But comfy-dev's ground-truth A/B: MoE wins (faster + dual-use as general LLM + slight fine-detail edge), and the subtle-proportion tier is below ALL VLs. Don't re-propose a dense-VL judge swap. (auto-memory project-qwen3vl-dense-pilot)

  • [2026-06-15] MastMed prototype containerized + exposed publicly via cloudflared. Self-contained static SPA (bun→caddy multi-stage build), basic-auth gated (operator's gating call), ana-docker:8089, cloudflared quick tunnel. See in-flight for the ephemeral-URL caveat + named-tunnel upgrade path.

  • [2026-06-15] infra-ops bootstrapped onto corviduo-dev (operator-directed) — infra-ops now has SSH + NOPASSWD sudo there, OVERRIDING the fleet bootstrap script's deliberate partner-box exclusion (explicit host arg). corviduo config edits no longer need vh + a sudo password. (auto-memory reference_corviduo_dev_emergency_ops)

  • [2026-06-15] R18 clip+caption placement = ana-ml2 (operator: "ana-docker… ana-ml2 unless irv-ml1 has room"; irv-ml1 lacked free VRAM for the 30B-A3B captioner). Models staged + stub deployed; see in-flight + the hard real-voice gate.

  • [2026-06-14] LiteLLM infra-ops admin key provisioned (operator) — resolves the LiteLLM half of the credential-migration directive; use it for ALL gateway ops (NOT sk-corvid). Value at ~/.config/litellm/infra-ops-key (mode 600); gateway reachable directly from nh3-dev at 10.250.50.70:4000. (auto-memory reference_litellm_infra_ops_key)

  • [2026-06-14] ana-ml2 GPU-1 vision upgraded: Qwen3.5-9B → Qwen3.6-35B-A3B (official FP8), served under its TRUE name only. qwen36-vl replaces qwen35-vl on :8007 (a0fed13). The stale qwen3.5-9b-fp8 name is KILLED at vLLM AND the litellm gateway (404/400) — a model is NEVER aliased under a prior model's name (silent substitution = downstream footgun; operator directive). Consumer comfy-dev/arbo migrated; arbo vkeys → all-proxy-models; shared all-agents-local key repointed qwen3.5-9b-fp8 → qwen3.6-35b-a3b. GPU-1 rebalanced for the ~34 GB FP8 weights (granite 0.35→0.24/64K; embed/rerank 0.05→0.03, reclaimed ~4 GB util-waste). Validated: vision correct, 20-concurrent = no OOM. (auto-memory feedback_no_false_model_aliases)

  • [2026-06-14] NVFP4 was the lighter fit (~21 GB) but is BLOCKED on vLLM — FP8 is the working vision path. nvidia/Qwen3.6-35B-A3B-NVFP4 won't load: the ModelOpt-NVFP4-MoE loader errors on expert/lm_head scale keys across 0.19.1 (w2_input_scale) AND 0.22.0 (lm_head.input_scale, vllm #44081) — a pattern across modelopt NVFP4 MoEs. Revisit NVFP4 (frees ~13 GB on GPU 1) once fixed; the 21 GB checkpoint stays cached on ana-ml2. (SUPERSEDED 2026-06-16 — it loads on vLLM 0.23.0; qwen36 swapped to NVFP4. See the top of this section.)

  • [2026-06-14] llama-swap qwen3.5-9b GPU-0 pin DROPPED; GPU 0 reserved for a creative-writing model (pick DEFERRED by operator). Deep-research (this session) on big-fast-uncensored creative for a 96 GB Blackwell: GLM-Steam-106B-A12B (already in the llama-swap config — balanced default) vs TheDrummer/Behemoth-X-123B-v2 (prose-tier, tops UGI writing+willingness) vs XORTRON-123B (max willingness, weak prose); GGUF-on-llama-swap is the serving path. Tracking: this session + llama-swap config (GLM-Steam present, untracked by operator choice).

  • [2026-06-14] R16 splice-pivot yield probe executed (infra-ops ran the irv-ml1 inference for brokkr; brokkr owns design + analysis). See Current state. Tracking: althing thread 01KV010WGS…, gen_yield_probe.py in irv-ml1:~/r16-vmoan-harness.

  • [2026-06-14] STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials. Agents currently reuse the operator's PERSONAL creds for infra ops — vh Gitea admin via tea (used this session to mint a read:package token for ratatoskr, id 13, under vh), sk-corvid litellm master for vkey admin. Stand up service accounts (a claude-bot Gitea user + scoped tokens, a distinct litellm admin key); re-mint consumer creds under them; flag personal-cred fallbacks until done. (auto-memory project_migrate_infra_access_to_claude_credentials)

  • [2026-06-14] R16 vmoan inline-generation arc CLOSED — v1 at default decode (rep1.2/temp0.8) is the final Chatterbox-tag inline artifact. Operator's ear rejected every alternative: v2/v3 windowing (omission vs coherence-loss), v4 multi-tag (cohesion held but lost to capacity-competition), emergent inline-token modulation (degenerates, not modulates), and the gen-time decode-polish sweep (soft tamers cut the NVV itself — same omission family as v2; p0 baseline beat p1). All adapters v1–v4 + tokenizer.json.v3bak preserved on irv-ml1:~/r16-vmoan-harness. Likely-next direction (deferred, NOT formalized): generate→bin→splice + one-shot-clone NVV pipeline routing around the inline-coherence wall. Tracking: brokkr R16 journal + althing thread 01KV010WGSSMPWRNCPAGSPK15Y.

  • [2026-06-14] Arbo deploy pipeline fixed, hardened, and version-controlled. Prod rebuilt v0.11.1 → v0.11.6 backend; the webhook machinery (arbo-deploy.sh + arbo-webhook.py, :9009 HMAC listener) is now repo-tracked at stacks/arbo/ (was host-only = recoverability foot-gun). Deploy reaches gitea via the INTERNAL route (10.250.50.70:222) and restarts the engine ONLY on catalog/ changes (graphs/frontend per-request; warn on src/|Dockerfile only — pyproject/uv.lock churn every commit). Operator kept arbo stack ownership in eshpfi (not migrated to comfy-dev's repo). Secret + .env stay host-only. Tracking: 6d66bc2, 6e58e57, stacks/arbo/README Q5.

  • [2026-06-13] Arbo prod bearer auth turned OFF — WireGuard is the access boundary (operator decision; reverses ADR-0001's "closed the open-auth hole"). ENGINE_TOKEN must be ABSENT, not empty (empty-string still gates) — removed from BOTH the host .env AND the compose environment: injection line. Original token backed up at irv-ml1:/opt/docker/compose/arbo/.env.pre-auth-off.bak; comfy-dev updated their ADR-0001. Tracking: db97899 + playbooks/arbo-disable-engine-token.yaml.

  • [2026-06-13] Storetank image-models archive DECOMMISSIONED; arbo is the single live ComfyUI model tree (502 G). Curated /storetank/image-models/comfy (was 919 G, the native /opt/ComfyUI/models symlink target) → killed everything superseded by arbo's current gen (Hunyuan, WAN2.1, FLUX.1, Chroma, SD3.5, orphaned umt5+llava ≈ 739 G) + migrated the keepers (gen-agnostic utilities + the SDXL/Pony stack, 177 G) into /storetank/arbo/models (same-fs move, skip-existing protects prod). Tracking: docs/storetank-image-models-archive.md + docs/arbo-comfyui-model-catalog.md (1902425→5007ec1).

  • [2026-06-13] GRANITE_KEY provisioned to comfy-dev's nh3-dev dev env at ~/.arbo_granite_key (0600) for the hero gen+judge script — verbatim copy of the prod arbo-prompt-enhance vkey (now extended to reach BOTH granite-4.1-8b AND qwen3.5-9b-fp8); nothing minted. The vkey README's "granite-only" wording was stale → corrected (f32c6dd).

  • [2026-06-13] ana-ml2 upgraded Ada → dual RTX PRO 6000 Blackwell Max-Q (96 GB each, cc 12.0 / sm_120; was dual RTX 6000 Ada 48 GB / cc 8.9 — confirmed live via nvidia-smi). Unlocks NVFP4 (FP4 tensor cores) and doubles VRAM headroom. CLAUDE.md GPU-spec doc-fix LANDED 355a240 (operator). Tracking: 19a07b9/1e2a3a1 ("Blackwell 96GB").

  • [2026-06-13] NVFP4-W4A4 is infeasible for Granite — FP8 stays the Granite-on-Blackwell format. W4A4 collapses at 30k context, proven producer-independent (modelopt AND llm-compressor both clean-NONE from the same BF16 base + wikitext-2k calib). No 4-bit wins both axes: W4A4 = quality collapse; W4A16-NVFP4/AWQ = weight-only dequant → bf16 (no FP4-core speedup). 30B retired. (auto-memory reference_nvfp4_w4a4_granite_infeasible)

  • [2026-06-13] Qwen3.5-9B VL (FP8) deployed on ana-ml2 GPU 1 — qwen35-vl stack, :8007, gateway alias qwen3.5-9b-fp8. Pinned nightly digest, not :latest: the stable release quantizes the VL vision tower under --quantization fp8 → garbage vision (LM fine); the nightly correctly excludes it. Re-pin + drop the pin once that exclusion lands stable. (2e3dcc2)

  • [2026-06-13] comfyui 325 G model tree migrated worktank → /storetank/arbo (worktank 97% → 26%). arbo is the consuming app; overlay bind-mount via COMFYUI_MODELS_DIR. (38186be) (See the 2026-06-13 archive-decommission decision above — this tree later absorbed the storetank-archive keepers, reaching 502 G.)

  • [2026-06-13] GPU layout settled on the Blackwell box. GPU 0 held free for large-model hot-loads (llama-swap pinned, edf0f91); GPU 1 steady-tenant — granite 131k ctx, qwen 65k, embed/rerank/reward trio, ~3.5 GB free (1e2a3a1, 19a07b9; trio re-floored for 96 GB, 20×-parallel-stable). embed/rerank left at floor — long docs chunked BEFORE embedding. max-model-len is a ceiling not a reservation.

  • [2026-06-13] Prefix caching pinned explicit on granite + qwen — benched ~6.5× faster TTFT on a shared ~4.5k-token summarizer template; soft/evictable, neutral when prefixes don't repeat. vLLM v1 defaults it ON (granite) but the qwen nightly defaults OFF — pin both. (a9a2be7)

  • [2026-06-13] granite-4.1-8b listed as the always-available summarizer/classifier + a shared all-agents key minted (operator-directed). Global ~/.claude/CLAUDE.md Global-tools entry; key alias all-agents-local, scoped to the FREE local models only (granite + qwen-vision + embed/rerank, NOT paid GLM), internal-gateway-only, rotatable. (auto-memory reference_litellm_gateway)

  • [2026-06-13] arbo engine + frontend stack stood up (ADR-0001) — irv-ml1 co-located inference engine (ee57e69), python-based healthcheck (slim image, no curl/wget, bdb3312), frontend ro-mounted from the checkout (922e8ad, ADR-0001 D2).

  • [2026-06-11] GLM thinking inverted at the LiteLLM gateway (operator call): glm-5.1 defaults thinking-OFF; glm-5.1-reasoning = same z.ai upstream, thinking ON. Mechanism: litellm_params.extra_body:{thinking:{type:disabled}} — drop_params strips a top-level thinking/reasoning_effort but forwards extra_body verbatim to z.ai. Shared-gateway change. (95b2701, auto-memory reference_litellm_gateway)

  • [2026-06-11] pi coding agent installed on nh3-dev as a GLM 5.1 harness — @earendil-works/pi-coding-agent via bun (npm's global prefix is /usr → needs sudo, bun avoids it). Config ~/.pi/agent/models.json, launcher ~/.local/bin/glm.

  • [2026-06-11] z.ai web-tools (regin) = z.ai hosted MCP path, NOT the /paas/v4 Tool API. WORKS: MCP streamable-HTTP at https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp, Authorization: Bearer $ZAI_API_KEY (the MCP key, distinct from Z_AI_API_KEY the LLM key). Reference impl = Worldtree's Leif agent.

  • [2026-06-10] Mac Pro migration framed: workstation-only (M2 Ultra ARM, racked NH3 on-subnet); sidecars stay Linux. migration-plan.md. (See in-flight.)

  • [2026-06-10] Worldtree deployed-config propagation is infra-ops's OWNED lane (operator ruling). worldtree-dev pings the config delta pre-merge; infra-ops syncs config/*.yaml from MERGED canonical to the /opt/worldtree*/config bind-mounts on demo+personal. The v0.33.8 9-HOUR demo outage (a model_roles.yaml startup-dep that never reached the VM) is the failure mode this prevents. providers.yaml stays hand-tuned. corviduo emergency-ops = ssh vh@10.250.50.152, docker no-sudo. (auto-memory reference_worldtree_deploys_cicd, reference_corviduo_dev_emergency_ops)

  • [2026-06-09] LiteLLM scoped virtual keys issued to consumers (operator-authorized): brokkr-smithy (all-proxy-models), arbo-prompt-enhance (comfy-dev — granite, later extended to qwen-vision). Mint via /key/generate (master sk-corvid), scope-restricted + rotatable, value → 600 file never the bus. (auto-memory reference_litellm_gateway)

56 older entries archived to archival-memory.md.

Tried and abandoned

  • [2026-06-16] vLLM 0.23.0 breaks Mistral MULTIMODAL at startup (MistralCommonImageProcessor has no attribute fetch_images, #44911 regression ~0.22.1+; same class hit Mistral-3.1/ Devstral). The dummy-image profiling crashes. Workarounds: --limit-mm-per-prompt '{"image":0,"video":0}' loads text-only; for VISION, pin v0.22.0 (last pre-regression). Tradeoff: v0.22.0 buffers reasoning (doesn't stream reasoning_content cleanly); a newer build streams it but breaks vision — vision won (Mistral is the abliteration-baseline).

  • [2026-06-16] Mistral -reasoning TTFT variance = Triton kernel JIT-compile-during- inference, NOT reasoning time. jit_monitor logs the spike (_compute_slot_mapping_kernel, apply_token_bitmask_inplace_kernel ← the tool-call/structured-output path). vLLM's startup warmup covers only SOME shapes; a first-hit uncovered shape JITs a kernel (seconds), then CACHES it — warm until container recreate (the Triton cache is IN-container, no volume, so recreate = cold again). Secondary: GPU idle-down to P8/180 MHz between requests. Also: Mistral reasoning_effort is BINARY (none|high only — medium/low → HTTP 400).

  • [2026-06-16] Worldtree old-baseline instances need the FULL config set synced, not the changeset delta. Personal (a PRE-mistral baseline) crash-looped (41 restarts) on DanglingBindingError: role 'echo' binds missing catalog entry 'qwen3.6-35-a3b-heretic' — its model_roles.yaml (2026-06-10, never migrated) still bound the retired heretic that the synced v0.35.10 providers removed. The "model_roles unchanged — skip" was correct only for demo (already at the v0.35.9 baseline). Old-baseline → sync providers + model_roles + defaults.

  • [2026-06-16] vh is a Gitea USER, not an org → no scoped package-write for a service account. Gitea user-namespace packages are owner/admin-only, so claude-bot can't push to vh/arbo without being admin. The fix is converting vh → a corviduo ORG (tabled). Also: vh's tea token lacks write:admin — create gitea users via the server-side gitea admin user create CLI (docker exec -u git gitea …), not the admin API.

  • [2026-06-16] docker compose up -d on a multi-service vLLM stack recreates ALL its services (shared .env → all configs re-hash) → concurrent memory-profiling race ("No available memory for the cache blocks"). Target one: docker compose up -d <service>. (auto-memory feedback_compose_up_recreates_whole_stack)

  • [2026-06-15] A comfyui restart-loop with exit 0 + OOMKilled=false is NOT OOM — suspect a native SEGFAULT. docker inspect shows only the LAST exit code; a GPU OOM is a catchable RuntimeError: CUDA out of memory, NEVER Fatal Python error: Segmentation fault. Diagnostic: RestartCount>0 = same container restarted in-place → the docker json-log ACCUMULATES across all cycles, so grep the full docker logs for the crash signature + the lines right before each restart banner. comfyui's was at VAE decode (sampling completes, then segfault loading AutoencodingEngine) — gen-triggered but native, not VRAM. Suspects logged: torch/cu130 mismatch, an injected aimdo cuda-funchooks lib, the VAE.

  • [2026-06-15] NEVER blanket-kill althing-light-monitor by PID — they're per-handle and MANY agents run their own on the shared nh3-dev box (heid / ratatoskr-dev / worldtree-dev / bifrost-dev each have one). A kill $(pgrep -f light-monitor) downs four peers' monitors (near-miss caught by checking /proc/<pid>/environ ALTHING_HANDLE + cwd first). Recover your OWN via althing-cli stop-monitor (v0.14.1+); a raw &-backgrounded monitor is untracked (no harness fire-notification) — always arm via run_in_background. (extends the pkill-self-match lesson below)

  • [2026-06-15] Manual docker compose up on corviduo Worldtree DOWNGRADES to a stale local :latest → crash-loop (config/image mismatch). Compose image: is …/worldtree:latest, but local :latest lagged ~2wk (21c90d5eeb2d); CI/CD deploys NEWER SHA images, and a real pull 401s (daemon has no gitea-registry creds — that's CI/CD's). --pull never then used the stale latest → personal crash-looped on ConfigurationError: echo.default_model missing → ~2-3 min personal outage. RECOVER: retag :latest to the image a HEALTHY SIBLING runs (docker inspect worldtree-worldtree-api-1 --format '{{.Image}}' → docker tag …), then up -d --pull never; verify Application startup complete. Don't manually recreate Worldtree on corviduo without pinning the correct image — image+config are CI/CD-coupled. (auto-memory reference_corviduo_dev_emergency_ops)

  • [2026-06-15] .claude.json (Claude Code config) truncates on ENOSPC → invalid JSON → CC resets to defaults on next start. nh3-dev hit 100% disk (a peer's R19 dataset pull staged ~147G to local /home instead of the 7TB NAS); the in-flight .claude.json write was cut mid-key. Repair = strip the dangling token + close the open braces, validate json.load. CC keeps its OWN rolling backups at ~/.claude/backups/ + writes a .claude.json.corrupted.* snapshot when it self-detects. After ANY ENOSPC on a box running CC/agents: integrity-check .claude.json (json.load) AND the althing bus DB (PRAGMA integrity_check).

  • [2026-06-15] pkill -f althing-light-monitor SELF-MATCHES the killing shell (the pattern is in the command's own argv) → kills itself mid-run (exit 144/truncated output). Stop the light-monitor via althing-cli stop-monitor or a captured PID — never pkill -f <string-present-in-this-command>. The singleton lock can also RACE to 2 live monitors during re-arm churn; keep exactly one tracked (run_in_background) monitor, and a raw & monitor is untracked (no harness fire-notification — don't use it).

  • [2026-06-14] vLLM ModelOpt-NVFP4-MoE loader is broken for current multimodal MoEs. nvidia/Qwen3.6-35B-A3B-NVFP4 fails weight-load: KeyError: layers.0.mlp.experts.w2_input_scale on 0.19.1, lm_head.input_scale not registered on 0.22.0 (vllm #44081); same class hits Gemma-4 MoE / Qwen3-30B-A3B NVFP4. The arch + quant ARE recognized (gets past arch resolution + vision-processor load) — it's the per-expert/lm_head scale-key mapping. Don't chase nightlies; use official FP8 until fixed.

  • [2026-06-14] vLLM sampler-warmup OOMs on a shared GPU even when weights fit — it warms the sampler with max_num_seqs (default 1024) dummy requests, and a big vocab (Qwen3.6 = 248K) makes that a huge transient logits tensor. A vision endpoint doesn't need 1024-way concurrency: set --max-num-seqs 32. Separately, post-load ValueError: No available memory for the cache blocks means util is too thin (weights+activation+graph ate it) — for 34 GB FP8 weights, util ≥ ~0.45 to leave KV room.

  • [2026-06-14] Recreating multiple vLLM services concurrently races the memory-profiling assertion — AssertionError: Error in memory profiling. Initial free memory X / current Y … other processes … release GPU memory while vLLM is profiling. Recreate co-tenant vLLM services ONE AT A TIME (force-recreate one, wait healthy, next).

  • [2026-06-14] embed/rerank (0.6B) at util 0.05 reserve ~5.5 GB each — mostly util-reservation WASTE, not need. A 0.6B model needs ~1.2 GB weights + ~2.5 GB CUDA/torch context; util 0.03 (~3.6 GB) fits with room, reclaiming ~4 GB (vLLM reserves the util fraction regardless of actual KV; embedding models barely use KV). Real-need floor ~3 GB — don't go to 0.02.

  • [2026-06-14] Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP. gitea.phasefinal.com = public 38.120.12.44 (ana-srv1); gitea is a container on ana-docker, git-SSH 10.250.50.70:222 + HTTP :3000. A fleet host egressing to public :22 gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (git fetch times out under set -euo pipefail, aborts before reset). Bit irv-ml1's arbo deploy. :22 on 10.250.50.70 is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented docs/orientation.md (6e58e57).

  • [2026-06-14] Chatterbox-Turbo decode-knob foot-guns (R16 v1-polish + emergent probes): the turbo length cap is max_gen_len (default 1000) on t3.inference_turbo, NOT max_new_tokens — and tts_turbo.generate does NOT forward it (wrap inference_turbo to cap). rep_pen 2.0 / temp 0.5 BACKFIRES (degenerate 24 s run-on). Soft decode tamers cut the NVV ITSELF, not just the run-on tail (operator: "p1 trims the moaning too") — gen-time polish can't beat v1's defaults. Inline base-NVV tokens DEGENERATE (moan-cascade + gibberish), they don't modulate the surrounding words.

  • [2026-06-13] Loading an old LoRA adapter after a vocab bump fails on embedding size. The harness config + tokenizer.json are now at new_vocab_size=50279 (v4 multi-tag); the v1/v2/v3 adapters are 50277. To load v1 (the accepted artifact), set cfg.new_vocab_size=50277 before load_finetuned_engine_lora (else PeftModel state_dict size mismatch). tokenizer.json.v3bak is the 50277 tokenizer for a clean restore.

  • [2026-06-13] Heavy from-source compile (MAX_JOBS=128) on the shared PROD GPU box PINS it — load hit 187, prod vLLM restarted, killed an in-flight quant. ana-ml2 hosts live inference; never run a big build there at full parallelism. Cap MAX_JOBS≤32, build off-box, or cgroup-constrain.

  • [2026-06-13] --quantization fp8 on a VL model can quantize the VISION TOWER → garbage vision (Qwen3.5-VL on stable vLLM: gray-grid output; LM answers text fine, so it "looks" healthy). The nightly excludes the vision tower. Validate the VISION path on a quantized VLM, not just text — and pin the engine digest with the exclusion.

  • [2026-06-13] vLLM's --gpu-memory-utilization is checked against FREE VRAM at startup, not total — on a shared card, growing one service before trimming a co-tenant OOMs. Trim the shrinking service FIRST, then grow. Size to the FREE budget.

  • [2026-06-13] The vllm/vllm-openai entrypoint is already ["vllm","serve"] — compose command: supplies the model as the first POSITIONAL arg + flags; a second serve/--model X → "unrecognized arguments". Same-class: tee masks the real exit code (use >); HF datasets rejects bare wikitext (needs Salesforce/wikitext).

  • [2026-06-13] Chatterbox-Turbo LoRA finetune: the repo's setup.py loads the WRONG tokenizer — pulls gpt2-medium + a grapheme merge file instead of the chatterbox-turbo GPT2 tokenizer (vocab.json+merges.txt, len 50276). Fix = override + delete the grapheme tokenizer.json; [vmoan] → new_vocab_size 50277 (1-row resize), lora_r 64 / alpha 128, modules_to_save=[text_emb,text_head]. Unique-stem corpus collision needs {index}_{stem} IDs. (irv-ml1:~/r16-vmoan-harness)

  • [2026-06-11] A completion-poll while pgrep -f <scriptname> SELF-MATCHES its own remote shell argv — its own pgrep -f always finds itself → the loop never exits. Use a match pattern ABSENT from the poll command (the python stage, or a sentinel file), not the driver's own name.

58 older entries archived to archival-memory.md.

Reference in New Issue View Git Blame Copy Permalink