memory: snapshot — 2026-06-16 ana-ml2 dual-NVFP4 reshape (GPU0 Mistral Small 4 256K/v0.22.0-vision + GPU1 qwen36 FP8→NVFP4 + Selene FP8 judge + GPU1 grows) + NVFP4-MoE-loads-on-0.23.0 (supersedes blocked) + claude-bot service account (corviduo-org tabled) + arbo→comfy-dev ownership + gitea runner on irv-ml1 + Worldtree demo/personal capability-profile migration (pre-sync-first); lessons: vLLM-0.23-breaks-Mistral-vision (#44911), Mistral-TTFT=Triton-JIT-spikes, vh-user-not-org blocks scoped package-write, old-baseline-instances-need-full-config-set; archived the 2026-06-05/08 cluster (11 entries)
This commit is contained in:
@@ -363,6 +363,24 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
||||
- `[2026-06-04]` **phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer; granite-4-small retired** from llama-swap. 50K ctx + FP8 KV. (`40a374b`) [superseded by the 2026-06-05 granite-4.1-8b decision]
|
||||
_Archived 2026-06-14._
|
||||
|
||||
- `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons; Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid `12aa5a9`); volva.service was crash-looping 203/EXEC. (`6e2f80e`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer.** Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not Q4/AWQ** — primary workload is high-concurrency, where FP8 scales ~linearly (2010 tok/s @ C=32). vLLM `vllm-granite` :8004 GPU 1, official IBM compressed-tensors FP8. (Then on Ada; box has since gone Blackwell.) (`34a43a0`, auto-memory `reference_ana_ml2_vllm_granite`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-05]` **Langfuse v3 on ana-docker (:3001) as the gateway trace UI**; LiteLLM `success_callback:[langfuse]` live. Pretty traces + tok/s dashboard. NOT a prerequisite (spend_logs already capture tokens+latency). (`9171e6a`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-05]` **Ollama BANNED fleet-wide** (operator directive) — never stand one up; tear down any found; serve via llama-swap or vLLM. (auto-memory `feedback_avoid_ollama`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-05]` **ComfyUI / FLUX.2 work split to `~/development/comfy-dev`** (dedicated repo + agent). eshpfi keeps the `comfyui`/`arbo` stack compose; comfy-dev owns the model/workflow knowledge. (auto-memory `reference_irv_ml1_ampere_quant`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is the structured-output profile, ON HOLD, no live consumer). Bind-mount CONFIGS (providers.yaml, vh-owned) ARE infra-ops's to apply directly — only the app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
## Tried and abandoned (archived)
|
||||
|
||||
- `[2026-04-30]` task-board workflow with
|
||||
@@ -769,3 +787,18 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
||||
|
||||
- `[2026-06-04]` **Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF — the "headgun" lesson.** Serve each model's canonical `tokenizer.apply_chat_template`, not the bundled template (bundled ones corrupt baselines). Verify via vLLM `/tokenize`→`/detokenize`. (`90e08f0`/`27eb537`)
|
||||
_Archived 2026-06-14._
|
||||
|
||||
- `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()` requires torchcodec → dies AFTER separating** (0 stems, rc=1). Fix = pin `torch==torchaudio==2.4.1` + `UV_LINK_MODE=copy`. Validate the SAVE path, not just import + GPU inference, on a bleeding-edge torch.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — fills KV to the `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs AFTER weights+KV load (crash-loops). Fix: free co-tenant room OR `--enforce-eager`. FP8 single-stream is batch-1 GEMV (memory-bound) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres rows; clone a default + swap the measure. tok/s is the `outputTokensPerSecond` MEASURE (metrics-API/dashboard query time), not a per-generation field. langfuse-web needs `HOSTNAME=0.0.0.0`. Host 3000 is gitea's → langfuse on 3001.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh + corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are vh-owned, the `.env` is deploy-owned 600.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL`; defaults to claude-haiku in code, opt-in via config not `.env`. Inspect the live container env + vh-owned config files first.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
+141
-48
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-06-15_
|
||||
_Last updated: 2026-06-16_
|
||||
|
||||
## Repo purpose
|
||||
|
||||
@@ -99,24 +99,67 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-06-15:_
|
||||
_As of 2026-06-16:_
|
||||
|
||||
- **ana-ml2 GPU layout RESHAPED again (2026-06-15/16) — both cards now full
|
||||
with NVFP4 tenants.** **GPU 0 = Mistral Small 4** (`mistral-small-4` stack,
|
||||
`mistralai/Mistral-Small-4-119B-2603-NVFP4`, 119B/6.5B-active MoE, :8010,
|
||||
gateway `mistral-small-4` + `mistral-small-4-reasoning`@effort=high). Pinned
|
||||
**vLLM v0.22.0** — the LAST release with working Mistral *vision* (#44911
|
||||
`fetch_images` regression breaks it on 0.22.1+/0.23.0). Serves the full native
|
||||
**256K context** (max-model-len 262144, max-num-seqs 32 — fits the tight card,
|
||||
~5 GB free). Text + vision both work; reasoning via `reasoning_effort` (BINARY:
|
||||
none|high). Dedicated single-tenant; the operator's creative-writing model
|
||||
(abliteration planned → it succeeds llama-swap). **GPU 1 = qwen36 swapped
|
||||
FP8→NVFP4** (`nvidia/Qwen3.6-35B-A3B-NVFP4`, fp16 KV, util 0.34, :8007, gateway
|
||||
name `qwen3.6-35b-a3b` UNCHANGED + `-thinking` variant) — the ModelOpt NVFP4
|
||||
MoE LOADS on 0.23.0 now (the 2026-06-14 "blocked" finding is RESOLVED). +
|
||||
**granite restored** (0.34/131072) + **Selene FP8 judge added** (`selene-1-mini-8b`,
|
||||
AtlaAI Selene-1-Mini-Llama-3.1-8B dynamic fp8, util 0.17, :8011, ctx 32768) +
|
||||
embed/rerank/reward. ~5.6 GB free. Prefix-caching ON on all 4 generative.
|
||||
**llama-swap is DOWN** (decommissioned from GPU 0 for Mistral; its qwen GGUF
|
||||
consumers migrated to the gateway). (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`)
|
||||
|
||||
- **arbo engine builds handed to comfy-dev; Gitea Actions runner LIVE on irv-ml1.**
|
||||
Operator approved comfy-dev owning arbo engine deploys (`deploy-engine.sh`,
|
||||
build-local). Runner = `irv-ml1-arbo` (act_runner v0.2.13, **host-executor as
|
||||
lkraven**, repo-scoped to vh/arbo, labels `self-hosted`+`irv-ml1`, systemd unit
|
||||
`act_runner.service`, polls the INTERNAL gitea route). comfy-dev to commit
|
||||
`.gitea/workflows/deploy.yml` — Phase 1 = engine-only auto-deploy (catalog stays
|
||||
on the :9009 webhook), Phase 2 catalog cutover later. infra-ops = box-level
|
||||
escalation only (runner, the `package:write` PAT once corviduo-org exists, webhook,
|
||||
the arbo-deploy.sh HEAD-baseline restart-detection bug still open).
|
||||
|
||||
- **claude-bot Gitea service account STOOD UP** (credential-migration foundation) —
|
||||
user id=3, non-admin/scoped, token `arbo-ci`=write:package+read:repository, creds
|
||||
at `~/.config/claude-bot/{gitea-password,gitea-token}` (mode 600). Registry-push
|
||||
DEFERRED: vh is a USER not an org, so Gitea won't grant a scoped service account
|
||||
package-write to `vh/*`. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
|
||||
|
||||
- **Worldtree demo + personal MIGRATED to v0.35.10 capability-named profiles** —
|
||||
persona substrate moved qwen→Mistral (profiles `dialogue`/`classification`→
|
||||
mistral-small-4, `deep-reasoning`→mistral-small-4-reasoning), Domari→selene; qwen
|
||||
heretic/heavy retired. Both green. The win: future model swaps (the abliterated
|
||||
Mistral) are now bind-mount-only `providers.yaml` edits — no image/agent/coupling.
|
||||
|
||||
- **R18 clip+caption auto-assist models STAGED on ana-ml2** (brokkr/Eitri) — WhisperX + PANNs CNN14_DecisionLevelAtt + Qwen3-Omni-30B-A3B-Captioner at Eitri's `/srv/r18-clip-caption` path contract (symlink → /tank; dedicated `r18clip` user, `workspaces/` 0700). Stub-level offline smoke PASSED (lo-only netns); Eitri closed the scaffold deliverable. AWAITING: Eitri/brokkr's concrete model runners (replace the deterministic stubs) → re-smoke the real path; AND the **hard real-voice gate** — no real victim voice through the shared-box captioner until ephemeral-(b) controls (in-VRAM-only, no shared-disk persistence, access-logged + auditd stood up) are built AND explicitly confirmed to brokkr. (auto-memory `project-r18-clip-caption-staging`)
|
||||
|
||||
- **MastMed prototype LIVE + public** — gated static-SPA container on ana-docker:8089 (caddy `basic_auth`, user `mastmed`) fronted by a cloudflared **quick tunnel** (ephemeral `*.trycloudflare.com` — dies on tunnel-container restart → new URL). Swap to a named tunnel (operator's scoped CF token + hostname) for a stable URL anytime; container + auth unchanged.
|
||||
|
||||
- **comfyui (irv-ml1) crash-loop = a VAE-decode SEGFAULT, NOT OOM** (diagnosed 2026-06-15 → comfy-dev, thread `01KV6FHCJ9…`). Per cycle: startup OK → `got prompt` → sampling 8/8 OK → `Requested to load AutoencodingEngine` → `Fatal Python error: Segmentation fault` → restart → queued prompt re-runs → loop. SERVING now (:8188 200, loop stopped — no VAE-decode gen since 20:08:51Z); recurs on the next such gen. comfy-dev owns the fix (suspects: torch/cu130 mismatch + `comfy_kitchen` cuda backend disabled, the injected `aimdo` cuda-funchooks, or the VAE). Infra-side offered to chase what injects `aimdo`.
|
||||
- **comfyui (irv-ml1) VAE-decode SEGFAULT — RESOLVED 2026-06-15.** Root cause was
|
||||
`comfy_aimdo`'s cudaMalloc hooks conflicting with ComfyUI's load-bearing
|
||||
`--disable-cuda-malloc`: with the optimized backend disabled (cu129<cu130) but
|
||||
the hooks still installed, VAE-decode allocation hit the dead path → native
|
||||
segfault. NOT cu130 (that would re-enable the allocator we deliberately disable).
|
||||
comfy-dev self-fixed: patched `comfy_aimdo/control.py` `init()` → return False
|
||||
(aimdo was non-functional anyway), folded into reprovision-venv.sh (their commit
|
||||
d5a2233). Verified clean across zimage/flux/SDXL.
|
||||
|
||||
- **Arbo prod current + deploy loop closed.** Engine **v0.11.22** + catalog **v0.11.23** (curated `/workflows` footer info now LIVE; engine rebuilt-from-main 2026-06-15, comfy-dev re-applied catalog). Frontend rides the git mount. Auth is OFF
|
||||
(WireGuard boundary). Pipeline is ban-immune (internal gitea route) +
|
||||
restart-correct (catalog-only) + version-controlled (`stacks/arbo/`).
|
||||
OPEN: the gitea registry has **never** held an arbo image — every deploy
|
||||
is local-image-only on irv-ml1, a rollback SPOF. Operator may mint a vh
|
||||
`package:write` PAT to backfill; the internal-route fix made it
|
||||
non-blocking, so NOT done. The public-IP fail2ban ban on irv-ml1's egress
|
||||
(`38.120.94.3`) on the gitea host — operator was handling the unban, but
|
||||
the internal-route repoint made it MOOT; ban-cleanup is optional hygiene
|
||||
(status unverified).
|
||||
- **Arbo prod: engine at v0.11.38** (three engine-only fixes 2026-06-16 for granite
|
||||
IR-shape "AI unavailable" degradations: intent_summary default, bare-string→list
|
||||
coercion); now comfy-dev-owned (see the runner entry above). Registry still holds
|
||||
NO arbo image (local-build-only) — that gap closes when the corviduo-org +
|
||||
`package:write` token land (deferred). Auth OFF (WireGuard boundary).
|
||||
|
||||
- **R16 splice-pivot yield probe RAN 2026-06-14** (was "awaiting go"). Inline-gen
|
||||
arc stays CLOSED (v1 @ default decode); the splice pivot's cheap de-risk is now
|
||||
@@ -127,16 +170,6 @@ _As of 2026-06-15:_
|
||||
Brokkr notified. NEXT: operator ear-bin (usable/impure) → brokkr's yield% /
|
||||
survivorship / identity analysis. `gen_yield_probe.py` in `irv-ml1:~/r16-vmoan-harness`.
|
||||
|
||||
- **ana-ml2 GPU layout RESHAPED by the FP8 vision cutover** (2026-06-14, `a0fed13`;
|
||||
dual RTX PRO 6000 Blackwell Max-Q, 96 GB ea, cc 12.0). GPU 1 now: **Qwen3.6-35B-
|
||||
A3B-FP8 vision** (`qwen36-vl`, :8007, util 0.46 — replaced the Qwen3.5-9B) +
|
||||
granite (0.24 / 64K) + reward (0.10) + embed/rerank (0.03 ea) ≈ 90 GB, ~7.5 GB
|
||||
headroom (the OOM buffer — verified under 20-concurrent/endpoint load). **GPU 0
|
||||
now FULLY FREE** (llama-swap qwen3.5-9b pin dropped) — reserved for a big-fast-
|
||||
uncensored creative-writing model (pick DEFERRED, see Recent decisions). Vision
|
||||
served under its TRUE name only (`qwen3.6-35b-a3b`); the old `qwen3.5-9b-fp8`
|
||||
name is killed at vLLM + the gateway.
|
||||
|
||||
- **R17 v2 corpus characterization still on the local irv-ml1 branch, push
|
||||
HELD** (`r17-v2-characterization`). No-push rule AND ASR-content-exposure
|
||||
call on the intimate-audio batch. Awaiting brokkr-collect or explicit push
|
||||
@@ -173,6 +206,55 @@ _As of 2026-06-15:_
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-06-16]` **NVFP4 ModelOpt MoE now LOADS on vLLM 0.23.0 — SUPERSEDES the
|
||||
2026-06-14 "blocked" finding (#44081 fixed).** Confirmed the cached
|
||||
`nvidia/Qwen3.6-35B-A3B-NVFP4` loads clean + generates on 0.23.0; swapped prod
|
||||
qwen36 FP8→NVFP4 (~13 GB reclaimed on GPU 1, fp16 KV). Validated by brokkr (speed:
|
||||
faster single-stream, ≥parity batched) + comfy-dev (vision anatomy-judge A/B: PASS).
|
||||
One W4A4 tell: minor multi-step chained-reasoning slip — doesn't bite the vision
|
||||
judge. Mistral's NVFP4 is a SEPARATE compressed-tensors path (day-0), distinct from
|
||||
the ModelOpt one. (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`)
|
||||
|
||||
- `[2026-06-16]` **Mistral Small 4 = the GPU-0 tenant** (operator) — fills the slot
|
||||
the deferred creative-writing pick held, and supersedes llama-swap (now down).
|
||||
NVFP4 119B MoE, pinned **vLLM v0.22.0** for working vision (0.23.0 breaks it),
|
||||
256K context. Operator's creative-writing model; **abliteration planned** → then it
|
||||
becomes the uncensored-creative model the persona substrate needs. Reasoning via
|
||||
`reasoning_effort` (BINARY none|high — no medium). (tracking: `c77a9aa`/`9a49963`)
|
||||
|
||||
- `[2026-06-16]` **Selene judge restored on vLLM FP8, NOT NVFP4** — W4A4 is too
|
||||
aggressive for a precision judge validated at Q6_K; FP8 ≥ Q6_K fidelity, and Selene
|
||||
is text-only Llama-3.1 so dynamic `--quantization fp8` has no vision tower to noise.
|
||||
`selene-1-mini-8b` on GPU 1 :8011. Worldtree's Domari role repointed to it. (`c985ede`)
|
||||
|
||||
- `[2026-06-16]` **GPU-1 buffer put to work: selene 0.13→0.17 + qwen36 0.32→0.34**
|
||||
— the two KV-bound generative models that gained Worldtree consumers; granite + the
|
||||
pooling models under-use their util, so growing THEM is wasted. (`e124a2f`)
|
||||
|
||||
- `[2026-06-16]` **claude-bot Gitea service account stood up; corviduo-ORG conversion
|
||||
TABLED** (operator). claude-bot is the credential-migration foundation. The proper
|
||||
fix for scoped agent package-write is converting vh (a USER) → a `corviduo` ORG —
|
||||
blast radius assessed (org creation = free; arbo-first migration = cheap, no registry
|
||||
images; full fleet = incremental, ~8 active vh images / 50 repos) — but tabled for
|
||||
later. Registry-push deferred until then. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
|
||||
|
||||
- `[2026-06-16]` **arbo engine builds → comfy-dev OWNERSHIP** (operator approved) +
|
||||
Gitea Actions runner stood up on irv-ml1 (host-executor as lkraven). infra-ops keeps
|
||||
box-level escalation (runner, package:write PAT pending corviduo-org, webhook). See
|
||||
the runner entry in Current state. (tracking: this snapshot + arbo thread `01KV70AB2B4B…`)
|
||||
|
||||
- `[2026-06-16]` **Worldtree persona substrate migrated qwen→Mistral via capability-
|
||||
named profiles** (v0.35.8→v0.35.10; demo + personal). PRE-SYNC-FIRST is now the
|
||||
standing pattern for any providers.yaml/profile change: sync config to the bind-mount
|
||||
BEFORE the deploy image-swap restart, so new agents + new config land together (the
|
||||
v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread `01KV6X0M…`)
|
||||
|
||||
- `[2026-06-15]` **qwen36 thinking-split at the gateway** (operator) — base
|
||||
`qwen3.6-35b-a3b` = thinking-OFF default (the vision judge needs it), `…-a3b-thinking`
|
||||
= opt-in reasoning. One hybrid checkpoint + `--reasoning-parser qwen3`; mirrors the
|
||||
glm-5.1 split. The judge's enable_thinking=false is what makes the NVFP4 swap
|
||||
transparent. (`6de0844`)
|
||||
|
||||
- `[2026-06-15]` **arbo engine rebuilt v0.11.6 → v0.11.22 from main; catalog re-applied (v0.11.23) — curated `/workflows` footer live.** src/ schema changes (Model/Lora.summary, Workflow.blurb/keywords + the /workflows endpoint) need a MANUAL image rebuild (the webhook warns-only on `src/`/`Dockerfile`). Built from `/worktank/arbo/repo`, bumped `ARBO_IMAGE` in the host .env, recreated + verified healthy (auto-rollback armed, unneeded). comfy-dev's v0.11.20 catalog push had first crash-looped prod (old baked schema `extra=forbid` vs new git-mounted catalog fields → ValidationError); they reverted the catalog, so rebuild-from-main loaded clean, then they re-applied (v0.11.23). comfyui is a SEPARATE compose project — the rebuild didn't touch it.
|
||||
|
||||
- `[2026-06-15]` **althing-core refreshed to v0.14.1 box-wide on nh3-dev** (forseti + heid both routed it; central-sequenced per DevOps-delegation since `uv tool install --force` swaps the shared binary all handles use). Fixes the monitor non-truncating-lock bug so `stop-monitor`/`--stop` reliably recovers orphans. The editable install (`uv tool install --force --editable .`) covers every nh3-dev handle; `--version` trued up 0.14.0→0.14.1; verified `--stop` works against a real orphan.
|
||||
@@ -191,7 +273,7 @@ _As of 2026-06-15:_
|
||||
|
||||
- `[2026-06-14]` **ana-ml2 GPU-1 vision upgraded: Qwen3.5-9B → Qwen3.6-35B-A3B (official FP8), served under its TRUE name only.** `qwen36-vl` replaces `qwen35-vl` on :8007 (`a0fed13`). The stale `qwen3.5-9b-fp8` name is KILLED at vLLM AND the litellm gateway (404/400) — a model is NEVER aliased under a prior model's name (silent substitution = downstream footgun; operator directive). Consumer comfy-dev/arbo migrated; arbo vkeys → all-proxy-models; shared `all-agents-local` key repointed qwen3.5-9b-fp8 → qwen3.6-35b-a3b. GPU-1 rebalanced for the ~34 GB FP8 weights (granite 0.35→0.24/64K; embed/rerank 0.05→0.03, reclaimed ~4 GB util-waste). Validated: vision correct, 20-concurrent = no OOM. (auto-memory `feedback_no_false_model_aliases`)
|
||||
|
||||
- `[2026-06-14]` **NVFP4 was the lighter fit (~21 GB) but is BLOCKED on vLLM — FP8 is the working vision path.** `nvidia/Qwen3.6-35B-A3B-NVFP4` won't load: the ModelOpt-NVFP4-MoE loader errors on expert/lm_head scale keys across 0.19.1 (`w2_input_scale`) AND 0.22.0 (`lm_head.input_scale`, vllm #44081) — a pattern across modelopt NVFP4 MoEs. Revisit NVFP4 (frees ~13 GB on GPU 1) once fixed; the 21 GB checkpoint stays cached on ana-ml2.
|
||||
- `[2026-06-14]` **NVFP4 was the lighter fit (~21 GB) but is BLOCKED on vLLM — FP8 is the working vision path.** `nvidia/Qwen3.6-35B-A3B-NVFP4` won't load: the ModelOpt-NVFP4-MoE loader errors on expert/lm_head scale keys across 0.19.1 (`w2_input_scale`) AND 0.22.0 (`lm_head.input_scale`, vllm #44081) — a pattern across modelopt NVFP4 MoEs. Revisit NVFP4 (frees ~13 GB on GPU 1) once fixed; the 21 GB checkpoint stays cached on ana-ml2. **(SUPERSEDED 2026-06-16 — it loads on vLLM 0.23.0; qwen36 swapped to NVFP4. See the top of this section.)**
|
||||
|
||||
- `[2026-06-14]` **llama-swap qwen3.5-9b GPU-0 pin DROPPED; GPU 0 reserved for a creative-writing model (pick DEFERRED by operator).** Deep-research (this session) on big-fast-uncensored creative for a 96 GB Blackwell: **GLM-Steam-106B-A12B** (already in the llama-swap config — balanced default) vs **TheDrummer/Behemoth-X-123B-v2** (prose-tier, tops UGI writing+willingness) vs XORTRON-123B (max willingness, weak prose); GGUF-on-llama-swap is the serving path. Tracking: this session + llama-swap config (GLM-Steam present, `untracked by operator choice`).
|
||||
|
||||
@@ -237,22 +319,43 @@ _As of 2026-06-15:_
|
||||
|
||||
- `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr-smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev — granite, later extended to qwen-vision). Mint via `/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus. (auto-memory `reference_litellm_gateway`)
|
||||
|
||||
- `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons; Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid `12aa5a9`); volva.service was crash-looping 203/EXEC. (`6e2f80e`)
|
||||
|
||||
- `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer.** Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not Q4/AWQ** — primary workload is high-concurrency, where FP8 scales ~linearly (2010 tok/s @ C=32). vLLM `vllm-granite` :8004 GPU 1, official IBM compressed-tensors FP8. (Then on Ada; box has since gone Blackwell.) (`34a43a0`, auto-memory `reference_ana_ml2_vllm_granite`)
|
||||
|
||||
- `[2026-06-05]` **Langfuse v3 on ana-docker (:3001) as the gateway trace UI**; LiteLLM `success_callback:[langfuse]` live. Pretty traces + tok/s dashboard. NOT a prerequisite (spend_logs already capture tokens+latency). (`9171e6a`)
|
||||
|
||||
- `[2026-06-05]` **Ollama BANNED fleet-wide** (operator directive) — never stand one up; tear down any found; serve via llama-swap or vLLM. (auto-memory `feedback_avoid_ollama`)
|
||||
|
||||
- `[2026-06-05]` **ComfyUI / FLUX.2 work split to `~/development/comfy-dev`** (dedicated repo + agent). eshpfi keeps the `comfyui`/`arbo` stack compose; comfy-dev owns the model/workflow knowledge. (auto-memory `reference_irv_ml1_ampere_quant`)
|
||||
|
||||
- `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is the structured-output profile, ON HOLD, no live consumer). Bind-mount CONFIGS (providers.yaml, vh-owned) ARE infra-ops's to apply directly — only the app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`)
|
||||
|
||||
_50 older entries archived to archival-memory.md._
|
||||
_56 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-06-16]` **vLLM 0.23.0 breaks Mistral MULTIMODAL at startup** (`MistralCommonImageProcessor
|
||||
has no attribute fetch_images`, #44911 regression ~0.22.1+; same class hit Mistral-3.1/
|
||||
Devstral). The dummy-image profiling crashes. Workarounds: `--limit-mm-per-prompt
|
||||
'{"image":0,"video":0}'` loads text-only; for VISION, pin **v0.22.0** (last pre-regression).
|
||||
Tradeoff: v0.22.0 *buffers* reasoning (doesn't stream reasoning_content cleanly); a newer
|
||||
build streams it but breaks vision — vision won (Mistral is the abliteration-baseline).
|
||||
|
||||
- `[2026-06-16]` **Mistral `-reasoning` TTFT variance = Triton kernel JIT-compile-during-
|
||||
inference, NOT reasoning time.** `jit_monitor` logs the spike (`_compute_slot_mapping_kernel`,
|
||||
`apply_token_bitmask_inplace_kernel` ← the tool-call/structured-output path). vLLM's startup
|
||||
warmup covers only SOME shapes; a first-hit uncovered shape JITs a kernel (seconds), then
|
||||
CACHES it — warm until container recreate (the Triton cache is IN-container, no volume, so
|
||||
recreate = cold again). Secondary: GPU idle-down to P8/180 MHz between requests. Also:
|
||||
Mistral `reasoning_effort` is BINARY (none|high only — medium/low → HTTP 400).
|
||||
|
||||
- `[2026-06-16]` **Worldtree old-baseline instances need the FULL config set synced, not the
|
||||
changeset delta.** Personal (a PRE-mistral baseline) crash-looped (41 restarts) on
|
||||
`DanglingBindingError: role 'echo' binds missing catalog entry 'qwen3.6-35-a3b-heretic'` —
|
||||
its `model_roles.yaml` (2026-06-10, never migrated) still bound the retired heretic that the
|
||||
synced v0.35.10 providers removed. The "model_roles unchanged — skip" was correct only for
|
||||
demo (already at the v0.35.9 baseline). Old-baseline → sync providers + model_roles + defaults.
|
||||
|
||||
- `[2026-06-16]` **vh is a Gitea USER, not an org → no scoped package-write for a service
|
||||
account.** Gitea user-namespace packages are owner/admin-only, so claude-bot can't push to
|
||||
`vh/arbo` without being admin. The fix is converting vh → a `corviduo` ORG (tabled). Also:
|
||||
vh's `tea` token lacks `write:admin` — create gitea users via the server-side `gitea admin
|
||||
user create` CLI (`docker exec -u git gitea …`), not the admin API.
|
||||
|
||||
- `[2026-06-16]` **`docker compose up -d` on a multi-service vLLM stack recreates ALL its
|
||||
services** (shared `.env` → all configs re-hash) → concurrent memory-profiling race
|
||||
("No available memory for the cache blocks"). Target one: `docker compose up -d <service>`.
|
||||
(auto-memory `feedback_compose_up_recreates_whole_stack`)
|
||||
|
||||
- `[2026-06-15]` **A comfyui restart-loop with `exit 0` + `OOMKilled=false` is NOT OOM — suspect a native SEGFAULT.** `docker inspect` shows only the LAST exit code; a GPU OOM is a catchable `RuntimeError: CUDA out of memory`, NEVER `Fatal Python error: Segmentation fault`. Diagnostic: `RestartCount>0` = same container restarted in-place → the docker json-log ACCUMULATES across all cycles, so grep the full `docker logs` for the crash signature + the lines right before each restart banner. comfyui's was at VAE decode (sampling completes, then segfault loading `AutoencodingEngine`) — gen-triggered but native, not VRAM. Suspects logged: torch/cu130 mismatch, an injected `aimdo` cuda-funchooks lib, the VAE.
|
||||
|
||||
- `[2026-06-15]` **NEVER blanket-kill `althing-light-monitor` by PID** — they're per-handle and MANY agents run their own on the shared nh3-dev box (heid / ratatoskr-dev / worldtree-dev / bifrost-dev each have one). A `kill $(pgrep -f light-monitor)` downs four peers' monitors (near-miss caught by checking `/proc/<pid>/environ` `ALTHING_HANDLE` + cwd first). Recover your OWN via `althing-cli stop-monitor` (v0.14.1+); a raw `&`-backgrounded monitor is untracked (no harness fire-notification) — always arm via `run_in_background`. (extends the pkill-self-match lesson below)
|
||||
@@ -289,14 +392,4 @@ _50 older entries archived to archival-memory.md._
|
||||
|
||||
- `[2026-06-11]` **A completion-poll `while pgrep -f <scriptname>` SELF-MATCHES its own remote shell argv** — its own `pgrep -f` always finds itself → the loop never exits. Use a match pattern ABSENT from the poll command (the python stage, or a sentinel file), not the driver's own name.
|
||||
|
||||
- `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()` requires torchcodec → dies AFTER separating** (0 stems, rc=1). Fix = pin `torch==torchaudio==2.4.1` + `UV_LINK_MODE=copy`. Validate the SAVE path, not just import + GPU inference, on a bleeding-edge torch.
|
||||
|
||||
- `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — fills KV to the `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs AFTER weights+KV load (crash-loops). Fix: free co-tenant room OR `--enforce-eager`. FP8 single-stream is batch-1 GEMV (memory-bound) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`)
|
||||
|
||||
- `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres rows; clone a default + swap the measure. tok/s is the `outputTokensPerSecond` MEASURE (metrics-API/dashboard query time), not a per-generation field. langfuse-web needs `HOSTNAME=0.0.0.0`. Host 3000 is gitea's → langfuse on 3001.
|
||||
|
||||
- `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh + corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are vh-owned, the `.env` is deploy-owned 600.
|
||||
|
||||
- `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL`; defaults to claude-haiku in code, opt-in via config not `.env`. Inspect the live container env + vh-owned config files first.
|
||||
|
||||
_53 older entries archived to archival-memory.md._
|
||||
_58 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user