vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d710e56acae0e1e07e47cc9e8b119bfe60bcbca1
esh-pfi-infrastructure/persistent-memory.md
T
vh d710e56aca memory: migrate persistent-memory.md to two-tier index (53 detail files) 
Split the 53 over-threshold dated log entries (Recent decisions, Tried
and abandoned) into per-entry persistent-memory.d/<slug>.md detail files,
leaving one-line pointers in the index; the 10 short entries stay inline.
Startup index drops 60,527 -> 25,256 bytes (492 -> 270 lines); entry
bodies move verbatim to on-demand detail files, so a fresh session loads
~25 KB instead of ~60 KB and pulls a detail file only when its pointer is
relevant. Top matter (Repo purpose, Tools & conventions, Current state)
is unchanged; both archival back-references preserved.

CLAUDE.md persistent-memory section now documents the index<->detail read
discipline (read the index, pull details on demand, never bulk-read the
dir, commit both together).

Auto-archival still held every dated entry back (all <30 days old); the
July burst begins aging past the 30-day guard ~2026-07-31.
2026-07-15 13:33:42 -07:00

25 KiB
Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-07-15

Always check for /tmp/infra-ops-handoff.md — if it exists and its Written: stamp is under an hour old, read it (it carries the in-flight handoff from the previous session), then delete it. Older than an hour: stale — delete it unread.

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo Role CI status
vh/task-board MCP + web dashboard for assistant task state (port 7878) push-to-main → CI deploys (2026-04-29)
vh/vor Inquisitor UI sidecar (port 7879) push-to-main → CI deploys (2026-04-29)
vh/nevermore Twice-daily LLM-curated briefing (port 8181, replaces news-digest) push-to-main → CI deploys (2026-04-30)
vh/asset-engine Internal control plane over inference services (port 8200, LAN-direct) push-to-main → CI deploys (2026-05-12)
vh/althing Lean trusted inter-agent message bus — v2 "email model" (v2.0.0b2, 2026-07): per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API /owner/* / althing-mcp stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. per-box uv tool install (NOT CI-deploy); nh3-dev = the DEV box (editable install of ~/development/althing, gets new versions first); nh3-extdev a mesh peer (model B: althing-svc + shared /srv/althing)
vh/mead-hall Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) push-to-main → CI deploys (2026-05-16)
vh/skaldsong Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) push-to-main → CI deploys (2026-05-19)
vh/Worldtree Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. push-to-main → CI build-and-deploy (runner on ana-docker)
vh/yt-voice-clipper YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see docs/runbooks/ytvc-autodeploy.md
vh/arbo Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
model-training-forge (mtf-dev) Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward) training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

  • Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)

  • pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

  • Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.

  • Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

  • Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.

  • gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.

  • docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.

  • scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.

  • Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

As of 2026-07-15 — a long fleet-ops + infra session. Everything below either LANDED (committed/pushed) or is PARKED on a clean async watch. No blocking work in flight.

Landed this session (2026-07-14 -> 15):

  • Homepage revamp — the flat ~22-service "AI Systems" group split into a dedicated AI tab with 6 role-based groups (Inference / Eval & Retrieval / Gateways & Chat / Speech (TTS) / Audio Tools / Image & Media) + an AI - Dormant group for the 7 stopped stacks. Relabeled 22 live compose files (homepage.group=) via surgical live sed + canonical backfill; GPU0 seats recreated in the operator's VRAM order (char-rp -> gen -> char-rp-reasoning). Committed 569e1af, pushed. Homepage runs on esh-docker-vm :5100 (config configs/homepage/).
  • esh-docker-vm rescued + NFS hardened — uptime-kuma + paperless-web were wedged in the D-state/phantom class; only a host REBOOT cleared it (infra-ops sudo). NFS fstab properly fixed (21d9a07, pushed): added x-systemd.before=docker.service,x-systemd.mount-timeout=30 to the 4 10.0.50.50: lines — the prior _netdev,nofail+After=remote-fs.target was silently defeated by nofail. Playbook playbooks/fix-esh-nfs-boot-ordering.yaml updated. Also: mOrpheus killed (irv-ml1 docker compose down; backend OOM-looping on the packed 3090) and a 1.36 GB HA coredump cleared.
  • Home Assistant config repo — vh/home-assistant-config (private gitea) for the HA Container at 10.0.50.46:8123 (= homeassistant on esh-docker-vm, macvlan, config named-volume). git-in-place in /config (root/infra-ops) + scoped deploy key; local clone ~/development/home-assistant-config. UI-managed HA -> allowlist model (YAML + curated secret-free .storage subset; secrets/DB/coredump gitignored).
  • soong-lab library persistence (corviduo-dev) — SOONG_LAB_LIBRARY_DIR=/home/infra-ops/soong-lab-data/library added to soong-lab.env (mirrors PORTRAIT_DIR), soong-lab-studio.service restarted. Unblocked the operator (Sindra import -> library save survives redeploys).
  • char-rp-reasoning OOM rescue + tune — the NVFP4+MTP seat (GPU0) crash-looped on a solo restart (GPU0 too tight). Fixed: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True + REASONING_GPU_MEM_UTIL 0.39->0.38 + REASONING_MAX_MODEL_LEN 196608->153600 (150K). Now ~4.5 GB GPU0 headroom, healthy; brokkr's text battery passed.
  • Statusline overhaul (~/.claude/statusline-command.sh, NOT this repo) — git state (branch/dirty/unpushed), 🔔/🔕 althing-monitor-armed, project tag, absolute tokens, per-session cost (.cost.total_cost_usd), threshold-colored ctx% + rate %s (green<60 / yellow60-90 / red>90).

PARKED (operator-directed holds):

  • image-bench (qwen-image-bench, GPU1) eviction — PENDING post-bake. Operator directed a full arbo -> gen switch (comfy-dev executed it, LIVE in prod: DEFAULT_LLM_MODEL=gen + ARBO_VISION_MODEL=gen); image-judge now has ZERO arbo tenancy. Sequencing = short prod bake on gen (~few days) THEN evict -> reclaim ~30 GB GPU1. Keep vllm-qwen-image-bench WARM as the instant-revert backstop; comfy-dev watches NSFW-anatomy scoring + tight-grounding (flip ARBO_VISION_MODEL back on regression). On a clean bake: evict + ping comfy-dev. Fully tracked in auto-memory project_arbo_gen_switch_imagebench_evict.
  • granite right-size — un-started GPU1 win. granite (GPU1) is ~29 GB (util 0.34 -> ~16.5 GB oversized KV for an 8B summarizer). Right-sizing util 0.34 -> ~0.13 frees ~16 GB WHILE keeping the summarizer (fleet's dedicated summarizer/classifier; aliases granite-4.1-8b/summarizer/classifier). Independent of image-bench; operator was freeing GPU1 for a TBD "something else".

Open loops for the fresh session:

  • stacks/heretic2-charrp-reasoning/ is UNTRACKED in git (?? stacks/heretic2-charrp-reasoning/) — the char-rp-reasoning compose stack is deployed but never committed; commit it (canonical mirror of the live seat).
  • althing monitor — re-arm /althing:monitor in the fresh session (the wake-listener dies on /clear). Open watch: comfy-dev on the arbo bake (no news = clean).
  • Carried standing items (non-blocking): T1 SFT LoRA dormant (operator pivoted off-the-shelf); rotate the 5 rest-server backup creds (operator, offline); brokkr P00 handoff for the NVFP4 char-rp-reasoning seat (in live traffic ahead of it).

Recent decisions

  • [2026-07-15] arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake → persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md

  • [2026-07-15] esh-docker-vm NFS fstab fix = x-systemd.before=docker.service → persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md

  • [2026-07-15] Homepage AI-tab revamp — flat "AI Systems" group -> dedicated AI tab, 6 role-based groups + AI-Dormant; committed 569e1af, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.)

  • [2026-07-15] Home Assistant config repo created (vh/home-assistant-config, private). UI-managed HA -> allowlist model (YAML + curated secret-free .storage subset). git-in-place in /config on esh-docker-vm + scoped deploy key + local clone ~/development/home-assistant-config.

  • [2026-07-15] char-rp-reasoning OOM rescue — solo-restart on the packed GPU0 crash-looped; fixed via expandable_segments:True + util 0.39->0.38 + max-model-len 192K->150K. LESSON (Tried): max-model-len does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom now.

  • [2026-07-15] soong-lab SOONG_LAB_LIBRARY_DIR made persistent (corviduo-dev) — was on the redeploy-wiped code default; set to /home/infra-ops/soong-lab-data/library (mirrors PORTRAIT_DIR), restarted. Closed a queued no-rush item; unblocked the operator.

  • [2026-07-15] Statusline overhauled (~/.claude/statusline-command.sh) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost (.cost.total_cost_usd) / threshold-colored ctx+rate (green<60 / yellow60-90 / red>90).

  • [2026-07-14] NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned → persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md

  • [2026-07-14] NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP → persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md

  • [2026-07-14] Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion → persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md

  • [2026-07-14] char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) → persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md

  • [2026-07-14] soong-lab webhook auto-deploy real root cause = gitea webhook.ALLOWED_HOST_LIST → persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md

  • [2026-07-13] #355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN) → persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md

  • [2026-07-13] Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong) → persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md

  • [2026-07-13] WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONE → persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md

  • [2026-07-13] WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 206 → persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md

  • [2026-07-13] Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile uv sync → persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md`

  • [2026-07-13] Ledger tier-3 consumer ledger:miranda provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); → persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md

  • [2026-07-10] Heimdall grant: ratatoskr affect.full on PERSONAL Worldtree (operator-approved, worldtree-dev R34-v1 request) → persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md

  • [2026-07-10] ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, SageAttention preserved, crash-loop AVOIDED → persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md

  • [2026-07-10] ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it) → persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md

  • [2026-07-10] Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized) → persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md

  • [2026-07-09] Two parked items closed: phantom qwen3.6-35b-a3b alias VERIFIED already-gone; ana-docker docker log-cap SOLVED no-bounce → persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md

  • [2026-07-09] granite→gen memory_extractor bind host-synced on demo+personal Worldtree (Vuong-directed, #335 Slice-4) → persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md

  • [2026-07-09] mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into gateway-chat → persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md

  • [2026-07-09] granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4) → persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md

  • [2026-07-08] RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (3… → persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md

  • [2026-07-08] worldtree Mimir deploy-blocker resolved (mid-session): → persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md

  • [2026-07-08] OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training → persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md

  • [2026-07-08] DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call) → persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md

  • [2026-07-08] T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py → persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md

  • [2026-07-08] T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe → persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md

142 older entries archived to archival-memory.md.

Tried and abandoned

  • [2026-07-15] docker.service After=remote-fs.target does NOT wait for nofail NFS mounts → persistent-memory.d/2026-07-15-docker-service-after-remote-fs-target-does-not.md

  • [2026-07-15] The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT → persistent-memory.d/2026-07-15-the-esh-docker-vm-d-state-phantom-container.md

  • [2026-07-15] vLLM max-model-len does NOT free GPU VRAM → persistent-memory.d/2026-07-15-vllm-max-model-len-does-not-free-gpu.md

  • [2026-07-15] Claude Code statusline .cost.total_cost_usd is per-SESSION → persistent-memory.d/2026-07-15-claude-code-statusline-cost-total-cost-usd-is.md

  • [2026-07-14] MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround → persistent-memory.d/2026-07-14-mtp-on-modelopt-no-checkpoint-config-skips-the.md

  • [2026-07-14] AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof → persistent-memory.d/2026-07-14-aeon-s-working-nvfp4-mtp-rp-seat-was.md

  • [2026-07-14] NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept → persistent-memory.d/2026-07-14-nvfp4-llm-compressor-compressed-tensors-gives-no-batch.md

  • [2026-07-14] NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent → persistent-memory.d/2026-07-14-nvfp4-spike-built-the-full-mtp-serve-scaffolding.md

  • [2026-07-14] MTP graft via top-level mtp.* tensor names does NOT survive AutoModelForCausalLM.from_pretrained → persistent-memory.d/2026-07-14-mtp-graft-via-top-level-mtp-tensor-names.md

  • [2026-07-14] gitea "test-delivery 204" is NOT proof a webhook works → persistent-memory.d/2026-07-14-gitea-test-delivery-204-is-not-proof-a.md

  • [2026-07-13] Relaying a peer's diagnosis as fact without confirming it against raw data → persistent-memory.d/2026-07-13-relaying-a-peer-s-diagnosis-as-fact-without.md

  • [2026-07-13] althing-cli reply <THREAD_id> (thread id, not a MESSAGE id) → "unknown message_id"; and reply to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use post --to <peer>. Bit me several times this session.

  • [2026-07-09] FP8 breaks mOrpheus audio-token generation → persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md

  • [2026-07-09] vllm/vllm-openai:latest crashes on Ampere IMPORT — Blackwell-only kernels (oink/aiter, has_device_capability(100)) die during import on the 3090/A6000. Pin v0.23.0 on irv-ml1's Ampere GPUs. (vllm/vllm-omni:v0.18.0 has a different entrypoint — don't use it either.)

  • [2026-07-09] Per-frame CPU SNAC decode is too slow for streaming — per-call overhead × ~60 frames serialized → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = windowed chunk decode (every 6 frames decode a [2 ctx | 6 | 2 ctx] window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).

  • [2026-07-09] Sentence-chunking TTS loses prosody → persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md

  • [2026-07-09] HF whisper datasets aren't actually whispered → persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md

  • [2026-07-08] Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE → persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md

  • [2026-07-08] Mistral3 + vLLM tokenizer/vision traps (serve MS3.2-24b, vLLM 0.24) → persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md

  • [2026-07-08] Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base → persistent-memory.d/2026-07-08-pantheon-reasoning-27b-refuses-dark-fiction-despite-an.md

  • [2026-07-08] Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance → persistent-memory.d/2026-07-08-pantheon-27b-mtp-on-vllm-compressed-tensors-0.md

  • [2026-07-07] vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639) → persistent-memory.d/2026-07-07-vllm-0-24-0-qwen3-5-lora-application.md

  • [2026-07-07] SGLang generic image can't LOAD our NVFP4 AEON → persistent-memory.d/2026-07-07-sglang-generic-image-can-t-load-our-nvfp4.md

  • [2026-07-07] SGLang --lora-target-modules CLI enum REJECTS the GDN names its own resolver asks for → persistent-memory.d/2026-07-07-sglang-lora-target-modules-cli-enum-rejects-the.md

  • [2026-07-07] Engine invocation footguns cost several wasted serve-bounces this session → persistent-memory.d/2026-07-07-engine-invocation-footguns-cost-several-wasted-serve-bounces.md

  • [2026-07-04] LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge → persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md

  • [2026-07-04] A systemd --user daemon that shells out to ~/.cargo/bin/~/.local/bin tools needs an explicit Environment=PATH → persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md

  • [2026-07-04] On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS → persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md

  • [2026-07-01] A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" is usually the pull-only-vs-build RACE, not registry-auth → persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md

  • [2026-07-01] MTP/spec-decode on a SHARED serving model helps single-stream but HURTS moderate-concurrency aggregate + silently ignores min_p/logit_bias (qwopus gen: N=1 +12%, N=4 −20%). Reserve for dedicated/interactive deployments.

  • [2026-07-02] irv-ml1 /worktank ROOT is root-owned — lkraven can't write there (irv-ml1 sudo needs a password) → stage model pulls to /home. PIN THE A6000 BY UUID for training (native-CUDA ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). CUDA_VISIBLE_DEVICES=GPU-<uuid>.

101 older entries archived to archival-memory.md.

Reference in New Issue View Git Blame Copy Permalink