memory: snapshot — 2026-06-16 litellm strip_empty_tools hook (d1bea13) + single-file gateway-chat.html playground (984ca3d) + claude-bot ADMIN on vh/arbo (arbo CI/CD via service account) + LitBench-RM reward judge served on irv-ml1 A6000 then taken to on-demand (held comfyui's slot) + #295 recall root-cause FLIPPED (score_breakdown-shape DISPROVEN → cold-recall agent_self scope axis vs ratatoskr conjunctive INV-005; Worldtree #297); lessons: litellm-500-Router.acompletion-missing-messages = a request missing Content-Type (NOT a gateway outage — cost 4 needless restarts), litellm-admin-UI-playground-cant-test-vLLM (#6228 empty-tools, proxy-hook-cant-reach-in-process-call), gitea-run-looks-like-never-fired-but-fired-then-skipped/failed-fast (check run list not runner). Archived the 06-09→06-13 cluster (24 entries: 17 decisions + 7 foot-guns).
This commit is contained in:
@@ -381,6 +381,57 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
||||
- `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is the structured-output profile, ON HOLD, no live consumer). Bind-mount CONFIGS (providers.yaml, vh-owned) ARE infra-ops's to apply directly — only the app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **Arbo prod bearer auth turned OFF — WireGuard is the access boundary** (operator decision; reverses ADR-0001's "closed the open-auth hole"). ENGINE_TOKEN must be **ABSENT, not empty** (empty-string still gates) — removed from BOTH the host `.env` AND the compose `environment:` injection line. Original token backed up at `irv-ml1:/opt/docker/compose/arbo/.env.pre-auth-off.bak`; comfy-dev updated their ADR-0001. Tracking: `db97899` + `playbooks/arbo-disable-engine-token.yaml`.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **Storetank image-models archive DECOMMISSIONED; arbo is the single live ComfyUI model tree (502 G).** Curated `/storetank/image-models/comfy` (was 919 G, the native `/opt/ComfyUI/models` symlink target) → killed everything superseded by arbo's current gen (Hunyuan, WAN2.1, FLUX.1, Chroma, SD3.5, orphaned umt5+llava ≈ 739 G) + migrated the keepers (gen-agnostic utilities + the SDXL/Pony stack, 177 G) into `/storetank/arbo/models` (same-fs move, skip-existing protects prod). Tracking: `docs/storetank-image-models-archive.md` + `docs/arbo-comfyui-model-catalog.md` (`1902425`→`5007ec1`).
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **GRANITE_KEY provisioned to comfy-dev's nh3-dev dev env** at `~/.arbo_granite_key` (0600) for the hero gen+judge script — verbatim copy of the prod `arbo-prompt-enhance` vkey (now extended to reach BOTH `granite-4.1-8b` AND `qwen3.5-9b-fp8`); nothing minted. The vkey README's "granite-only" wording was stale → corrected (`f32c6dd`).
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **ana-ml2 upgraded Ada → dual RTX PRO 6000 Blackwell Max-Q** (96 GB each, cc 12.0 / sm_120; was dual RTX 6000 Ada 48 GB / cc 8.9 — confirmed live via `nvidia-smi`). Unlocks NVFP4 (FP4 tensor cores) and doubles VRAM headroom. CLAUDE.md GPU-spec doc-fix **LANDED `355a240`** (operator). Tracking: `19a07b9`/`1e2a3a1` ("Blackwell 96GB").
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **NVFP4-W4A4 is infeasible for Granite — FP8 stays the Granite-on-Blackwell format.** W4A4 collapses at 30k context, proven **producer-independent** (modelopt AND llm-compressor both clean-NONE from the same BF16 base + wikitext-2k calib). No 4-bit wins both axes: W4A4 = quality collapse; W4A16-NVFP4/AWQ = weight-only dequant → bf16 (no FP4-core speedup). **30B retired**. (auto-memory `reference_nvfp4_w4a4_granite_infeasible`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **Qwen3.5-9B VL (FP8) deployed on ana-ml2 GPU 1** — `qwen35-vl` stack, :8007, gateway alias `qwen3.5-9b-fp8`. **Pinned nightly digest, not `:latest`**: the stable release quantizes the VL *vision tower* under `--quantization fp8` → garbage vision (LM fine); the nightly correctly excludes it. Re-pin + drop the pin once that exclusion lands stable. (`2e3dcc2`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **comfyui 325 G model tree migrated worktank → `/storetank/arbo`** (worktank 97% → 26%). `arbo` is the consuming app; overlay bind-mount via `COMFYUI_MODELS_DIR`. (`38186be`) (See the 2026-06-13 archive-decommission decision above — this tree later absorbed the storetank-archive keepers, reaching 502 G.)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **GPU layout settled on the Blackwell box.** GPU 0 held free for large-model hot-loads (llama-swap pinned, `edf0f91`); GPU 1 steady-tenant — granite 131k ctx, qwen 65k, embed/rerank/reward trio, ~3.5 GB free (`1e2a3a1`, `19a07b9`; trio re-floored for 96 GB, 20×-parallel-stable). embed/rerank left at floor — long docs chunked BEFORE embedding. max-model-len is a ceiling not a reservation.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **Prefix caching pinned explicit on granite + qwen** — benched ~6.5× faster TTFT on a shared ~4.5k-token summarizer template; soft/evictable, neutral when prefixes don't repeat. vLLM v1 defaults it ON (granite) but the qwen nightly defaults OFF — pin both. (`a9a2be7`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **granite-4.1-8b listed as the always-available summarizer/classifier + a shared all-agents key minted** (operator-directed). Global `~/.claude/CLAUDE.md` Global-tools entry; key alias `all-agents-local`, scoped to the FREE local models only (granite + qwen-vision + embed/rerank, NOT paid GLM), internal-gateway-only, rotatable. (auto-memory `reference_litellm_gateway`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **arbo engine + frontend stack stood up** (ADR-0001) — irv-ml1 co-located inference engine (`ee57e69`), python-based healthcheck (slim image, no curl/wget, `bdb3312`), frontend ro-mounted from the checkout (`922e8ad`, ADR-0001 D2).
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-11]` **GLM thinking inverted at the LiteLLM gateway** (operator call): `glm-5.1` defaults thinking-OFF; `glm-5.1-reasoning` = same z.ai upstream, thinking ON. Mechanism: `litellm_params.extra_body:{thinking:{type:disabled}}` — `drop_params` strips a top-level `thinking`/`reasoning_effort` but forwards `extra_body` verbatim to z.ai. Shared-gateway change. (`95b2701`, auto-memory `reference_litellm_gateway`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-11]` **pi coding agent installed on nh3-dev as a GLM 5.1 harness** — `@earendil-works/pi-coding-agent` via **bun** (npm's global prefix is `/usr` → needs sudo, bun avoids it). Config `~/.pi/agent/models.json`, launcher `~/.local/bin/glm`.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-11]` **z.ai web-tools (regin) = z.ai hosted MCP path, NOT the `/paas/v4` Tool API.** WORKS: MCP streamable-HTTP at `https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp`, `Authorization: Bearer $ZAI_API_KEY` (the **MCP** key, distinct from `Z_AI_API_KEY` the LLM key). Reference impl = Worldtree's Leif agent.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-10]` **Mac Pro migration framed: workstation-only** (M2 Ultra ARM, racked NH3 on-subnet); sidecars stay Linux. `migration-plan.md`. (See in-flight.)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-10]` **Worldtree deployed-config propagation is infra-ops's OWNED lane** (operator ruling). worldtree-dev pings the config delta pre-merge; infra-ops syncs `config/*.yaml` from MERGED canonical to the `/opt/worldtree*/config` bind-mounts on demo+personal. The v0.33.8 9-HOUR demo outage (a `model_roles.yaml` startup-dep that never reached the VM) is the failure mode this prevents. providers.yaml stays hand-tuned. corviduo emergency-ops = `ssh vh@10.250.50.152`, docker no-sudo. (auto-memory `reference_worldtree_deploys_cicd`, `reference_corviduo_dev_emergency_ops`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr-smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev — granite, later extended to qwen-vision). Mint via `/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus. (auto-memory `reference_litellm_gateway`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
## Tried and abandoned (archived)
|
||||
|
||||
- `[2026-04-30]` task-board workflow with
|
||||
@@ -802,3 +853,24 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
||||
|
||||
- `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL`; defaults to claude-haiku in code, opt-in via config not `.env`. Inspect the live container env + vh-owned config files first.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **Loading an old LoRA adapter after a vocab bump fails on embedding size.** The harness config + `tokenizer.json` are now at `new_vocab_size=50279` (v4 multi-tag); the v1/v2/v3 adapters are 50277. To load v1 (the accepted artifact), set `cfg.new_vocab_size=50277` before `load_finetuned_engine_lora` (else PeftModel state_dict size mismatch). `tokenizer.json.v3bak` is the 50277 tokenizer for a clean restore.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **Heavy from-source compile (`MAX_JOBS=128`) on the shared PROD GPU box PINS it** — load hit 187, prod vLLM restarted, killed an in-flight quant. ana-ml2 hosts live inference; never run a big build there at full parallelism. Cap `MAX_JOBS≤32`, build off-box, or cgroup-constrain.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **`--quantization fp8` on a VL model can quantize the VISION TOWER → garbage vision** (Qwen3.5-VL on stable vLLM: gray-grid output; LM answers text fine, so it "looks" healthy). The nightly excludes the vision tower. Validate the VISION path on a quantized VLM, not just text — and pin the engine digest with the exclusion.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **vLLM's `--gpu-memory-utilization` is checked against FREE VRAM at startup, not total** — on a shared card, growing one service before trimming a co-tenant OOMs. Trim the shrinking service FIRST, then grow. Size to the FREE budget.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **The `vllm/vllm-openai` entrypoint is already `["vllm","serve"]`** — compose `command:` supplies the model as the first POSITIONAL arg + flags; a second `serve`/`--model X` → "unrecognized arguments". Same-class: `tee` masks the real exit code (use `>`); HF `datasets` rejects bare `wikitext` (needs `Salesforce/wikitext`).
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-13]` **Chatterbox-Turbo LoRA finetune: the repo's `setup.py` loads the WRONG tokenizer** — pulls gpt2-medium + a grapheme merge file instead of the chatterbox-turbo GPT2 tokenizer (vocab.json+merges.txt, len 50276). Fix = override + delete the grapheme `tokenizer.json`; `[vmoan]` → new_vocab_size 50277 (1-row resize), lora_r 64 / alpha 128, modules_to_save=[text_emb,text_head]. Unique-stem corpus collision needs `{index}_{stem}` IDs. (`irv-ml1:~/r16-vmoan-harness`)
|
||||
_Archived 2026-06-16._
|
||||
|
||||
- `[2026-06-11]` **A completion-poll `while pgrep -f <scriptname>` SELF-MATCHES its own remote shell argv** — its own `pgrep -f` always finds itself → the loop never exits. Use a match pattern ABSENT from the poll command (the python stage, or a sentinel file), not the driver's own name.
|
||||
_Archived 2026-06-16._
|
||||
|
||||
+20
-52
@@ -101,7 +101,11 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
_As of 2026-06-16:_
|
||||
|
||||
- **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven — persist+dispatch GREEN, recall-injection OPEN (handed off, OUT of loop per operator).** The missing server-side step was the per-PORT url-guard allowlist: added `10.100.10.50:8391` to personal's `BIFROST_CLIENT_ALLOWED_HOSTS` (affect `:8390` was already in; via `playbooks/wire-personal-worldtree-memory-allowlist.yaml`, c99aa49). Consumer key REUSED (plane-agnostic, no mint). Drove define→bind→persist(async `upsert_many` 200)→recall(`search` 200); ratatoskr's `memory.db` durable + 1024-dim indexed. **Promotion gate = a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), NOT `agent_self_enabled`.** Cross-session recall-INJECTION root-caused to a SCORE-SHAPE seam: Worldtree's injector reads `score_breakdown.{similarity,salience}`, but ratatoskr's `search()` returns top-level `score` + `chunk.salience` (0.9, ≥0.85) → both gate-reads collapse to 0 → filtered AFTER a clean 200. Handed to worldtree-dev (#295 recall leg OPEN; their ADR-0009 wire-spec call — bifrost returns `score_breakdown`, or Worldtree reads existing fields). (auto-memory `reference_bifrost_plane_wiring`)
|
||||
- **LitBench-RM stood up → now ON-DEMAND / DOWN.** `SAA-Lab/Llama8B-CreativeWritingVerifier` (R19 creative-quality reward judge for brokkr/Dvalin) served on irv-ml1 A6000 via vLLM `--runner pooling` (`http://10.100.79.3:8202/classify`, raw passage text → scalar). brokkr-validated (does NOT penalize explicit content). Taken DOWN to on-demand (operator) — resident it held ~19.6 GB crowding comfyui's A6000 slot; weights staged on irv-ml1, ~90s respin (command in auto-memory). (auto-memory `reference_litbench_rm_irv_ml1`)
|
||||
|
||||
- **litellm gateway healthy; `strip_empty_tools` hook + `gateway-chat.html` shipped** (see Recent decisions). **arbo + brokkr litellm keys are already `all-proxy-models`** — verified, so "grant arbo/brokkr full access" was already done. claude-bot now has **admin on vh/arbo**; **arbo CI/CD Phase 1 is firing** (runner live; comfy-dev owns `deploy.yml`, fixed the change-detection in `9eb743a` — run #4 then failed-fast on their side, reading UI logs).
|
||||
|
||||
- **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven — persist+dispatch GREEN, recall-injection OPEN (handed off, OUT of loop per operator).** The missing server-side step was the per-PORT url-guard allowlist: added `10.100.10.50:8391` to personal's `BIFROST_CLIENT_ALLOWED_HOSTS` (affect `:8390` was already in; via `playbooks/wire-personal-worldtree-memory-allowlist.yaml`, c99aa49). Consumer key REUSED (plane-agnostic, no mint). Drove define→bind→persist(async `upsert_many` 200)→recall(`search` 200); ratatoskr's `memory.db` durable + 1024-dim indexed. **Promotion gate = a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), NOT `agent_self_enabled`.** Cross-session recall-INJECTION root-caused to a SCORE-SHAPE seam: Worldtree's injector reads `score_breakdown.{similarity,salience}`, but ratatoskr's `search()` returns top-level `score` + `chunk.salience` (0.9, ≥0.85) → both gate-reads collapse to 0 → filtered AFTER a clean 200. Handed to worldtree-dev (#295 recall leg OPEN; their ADR-0009 wire-spec call — bifrost returns `score_breakdown`, or Worldtree reads existing fields). (auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED →** worldtree-dev's re-triage + my `:8391`-log pull DISPROVED the score_breakdown-shape theory: the wire shape is FINE (Worldtree *synthesizes* `score_breakdown` from the `score`+`chunk.salience` ratatoskr emits). Real cause = the COLD cross-session recall sends `scope_filter` with an extra `agent_self` axis → ratatoskr's conjunctive INV-005 `_scope_matches` drops the `{end_user}`-only chunks → 0 hits. Now Worldtree-side, tracked **Worldtree #297**. (auto-memory `project_worldtree_295_recall_rootcause`)
|
||||
|
||||
- **Worldtree demo + personal `character` model = mistral-small-4** (flipped from qwen3.6-35-a3b, 2026-06-16) — reordered `model_roles.yaml` `character.binds` mistral-first (first bind = default), qwen kept in the switch-allowlist; applied via pin-safe recreate, fresh-agent resolution verified. Backups `model_roles.yaml.bak-pre-mistral-character`.
|
||||
|
||||
@@ -210,7 +214,13 @@ _As of 2026-06-16:_
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-06-16]` **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven.** The missing server-side step was the per-PORT `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist (`:8391`; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the `score_breakdown` shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), not `agent_self_enabled`. (tracking: c99aa49 + auto-memory `reference_bifrost_plane_wiring`)
|
||||
- `[2026-06-16]` **litellm `strip_empty_tools` pre-call hook shipped** (`d1bea13`) — an empty `tools:[]` 500s vLLM ("tools must not be an empty array"); a global `litellm_settings.callbacks` CustomLogger pops it (+ orphaned `tool_choice`) before forwarding, so it covers EVERY vLLM model, not one. `drop_params` only drops unsupported PARAMS, not empty VALUES. Mounts beside config.yaml (litellm resolves callbacks relative to the config dir). Verified live across granite/mistral/stream. (`stacks/litellm/conf/strip_empty_tools.py`)
|
||||
|
||||
- `[2026-06-16]` **single-file `gateway-chat.html` playground shipped** (`984ca3d`, `tools/`) — zero-dep browser chat straight to the gateway (`:4000`, CORS open), system-prompt box, streaming SSE, renders `reasoning_content`, NEVER sends `tools`. Built because the LiteLLM admin-UI playground can't test vLLM-backed models (see Tried-and-abandoned). Serve on-request via `python3 -m http.server -d tools`.
|
||||
|
||||
- `[2026-06-16]` **claude-bot granted ADMIN on vh/arbo** (operator) — arbo CI/CD gitea ops now route through the service account, not the operator's personal `vh` (the credential-migration directive). Operator added claude-bot as Administrator collaborator; claude-bot self-serves via its `gitea-password` (basic-auth = FULL perms; its stored token is package-write-only, can't change repo settings). (auto-memory `reference_claude_bot_gitea_creds`)
|
||||
|
||||
- `[2026-06-16]` **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven.** The missing server-side step was the per-PORT `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist (`:8391`; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the `score_breakdown` shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), not `agent_self_enabled`. (tracking: c99aa49 + auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED:** the `score_breakdown`-shape root-cause was disproven; real cause = cold-recall `scope_filter` adds an `agent_self` axis → ratatoskr's conjunctive INV-005 match drops the `{end_user}`-only chunks → 0 hits. Now Worldtree #297. (auto-memory `project_worldtree_295_recall_rootcause`)
|
||||
|
||||
- `[2026-06-16]` **infra-ops = the DURABLE admin path on corviduo-dev** (operator granted standing admin; extends the 2026-06-15 bootstrap) — added a `Host corviduo-dev → User infra-ops` `~/.ssh/config` alias + repointed `servers/corviduo-dev/ssh-target` to `infra-ops@`; `ssh corviduo-dev` / `elway corviduo-dev` now land as infra-ops w/ NOPASSWD (vh = explicit-only human path). elway resolves via ssh-config, NOT the ssh-target file. (tracking: c99aa49 + auto-memory `reference_corviduo_dev_emergency_ops`)
|
||||
|
||||
@@ -295,44 +305,16 @@ _As of 2026-06-16:_
|
||||
|
||||
- `[2026-06-14]` **Arbo deploy pipeline fixed, hardened, and version-controlled.** Prod rebuilt v0.11.1 → **v0.11.6** backend; the webhook machinery (`arbo-deploy.sh` + `arbo-webhook.py`, :9009 HMAC listener) is now repo-tracked at `stacks/arbo/` (was host-only = recoverability foot-gun). Deploy reaches gitea via the INTERNAL route (`10.250.50.70:222`) and restarts the engine ONLY on `catalog/` changes (graphs/frontend per-request; warn on `src/`|`Dockerfile` only — pyproject/uv.lock churn every commit). Operator kept arbo stack ownership in **eshpfi** (not migrated to comfy-dev's repo). Secret + `.env` stay host-only. Tracking: `6d66bc2`, `6e58e57`, `stacks/arbo/README` Q5.
|
||||
|
||||
- `[2026-06-13]` **Arbo prod bearer auth turned OFF — WireGuard is the access boundary** (operator decision; reverses ADR-0001's "closed the open-auth hole"). ENGINE_TOKEN must be **ABSENT, not empty** (empty-string still gates) — removed from BOTH the host `.env` AND the compose `environment:` injection line. Original token backed up at `irv-ml1:/opt/docker/compose/arbo/.env.pre-auth-off.bak`; comfy-dev updated their ADR-0001. Tracking: `db97899` + `playbooks/arbo-disable-engine-token.yaml`.
|
||||
|
||||
- `[2026-06-13]` **Storetank image-models archive DECOMMISSIONED; arbo is the single live ComfyUI model tree (502 G).** Curated `/storetank/image-models/comfy` (was 919 G, the native `/opt/ComfyUI/models` symlink target) → killed everything superseded by arbo's current gen (Hunyuan, WAN2.1, FLUX.1, Chroma, SD3.5, orphaned umt5+llava ≈ 739 G) + migrated the keepers (gen-agnostic utilities + the SDXL/Pony stack, 177 G) into `/storetank/arbo/models` (same-fs move, skip-existing protects prod). Tracking: `docs/storetank-image-models-archive.md` + `docs/arbo-comfyui-model-catalog.md` (`1902425`→`5007ec1`).
|
||||
|
||||
- `[2026-06-13]` **GRANITE_KEY provisioned to comfy-dev's nh3-dev dev env** at `~/.arbo_granite_key` (0600) for the hero gen+judge script — verbatim copy of the prod `arbo-prompt-enhance` vkey (now extended to reach BOTH `granite-4.1-8b` AND `qwen3.5-9b-fp8`); nothing minted. The vkey README's "granite-only" wording was stale → corrected (`f32c6dd`).
|
||||
|
||||
- `[2026-06-13]` **ana-ml2 upgraded Ada → dual RTX PRO 6000 Blackwell Max-Q** (96 GB each, cc 12.0 / sm_120; was dual RTX 6000 Ada 48 GB / cc 8.9 — confirmed live via `nvidia-smi`). Unlocks NVFP4 (FP4 tensor cores) and doubles VRAM headroom. CLAUDE.md GPU-spec doc-fix **LANDED `355a240`** (operator). Tracking: `19a07b9`/`1e2a3a1` ("Blackwell 96GB").
|
||||
|
||||
- `[2026-06-13]` **NVFP4-W4A4 is infeasible for Granite — FP8 stays the Granite-on-Blackwell format.** W4A4 collapses at 30k context, proven **producer-independent** (modelopt AND llm-compressor both clean-NONE from the same BF16 base + wikitext-2k calib). No 4-bit wins both axes: W4A4 = quality collapse; W4A16-NVFP4/AWQ = weight-only dequant → bf16 (no FP4-core speedup). **30B retired**. (auto-memory `reference_nvfp4_w4a4_granite_infeasible`)
|
||||
|
||||
- `[2026-06-13]` **Qwen3.5-9B VL (FP8) deployed on ana-ml2 GPU 1** — `qwen35-vl` stack, :8007, gateway alias `qwen3.5-9b-fp8`. **Pinned nightly digest, not `:latest`**: the stable release quantizes the VL *vision tower* under `--quantization fp8` → garbage vision (LM fine); the nightly correctly excludes it. Re-pin + drop the pin once that exclusion lands stable. (`2e3dcc2`)
|
||||
|
||||
- `[2026-06-13]` **comfyui 325 G model tree migrated worktank → `/storetank/arbo`** (worktank 97% → 26%). `arbo` is the consuming app; overlay bind-mount via `COMFYUI_MODELS_DIR`. (`38186be`) (See the 2026-06-13 archive-decommission decision above — this tree later absorbed the storetank-archive keepers, reaching 502 G.)
|
||||
|
||||
- `[2026-06-13]` **GPU layout settled on the Blackwell box.** GPU 0 held free for large-model hot-loads (llama-swap pinned, `edf0f91`); GPU 1 steady-tenant — granite 131k ctx, qwen 65k, embed/rerank/reward trio, ~3.5 GB free (`1e2a3a1`, `19a07b9`; trio re-floored for 96 GB, 20×-parallel-stable). embed/rerank left at floor — long docs chunked BEFORE embedding. max-model-len is a ceiling not a reservation.
|
||||
|
||||
- `[2026-06-13]` **Prefix caching pinned explicit on granite + qwen** — benched ~6.5× faster TTFT on a shared ~4.5k-token summarizer template; soft/evictable, neutral when prefixes don't repeat. vLLM v1 defaults it ON (granite) but the qwen nightly defaults OFF — pin both. (`a9a2be7`)
|
||||
|
||||
- `[2026-06-13]` **granite-4.1-8b listed as the always-available summarizer/classifier + a shared all-agents key minted** (operator-directed). Global `~/.claude/CLAUDE.md` Global-tools entry; key alias `all-agents-local`, scoped to the FREE local models only (granite + qwen-vision + embed/rerank, NOT paid GLM), internal-gateway-only, rotatable. (auto-memory `reference_litellm_gateway`)
|
||||
|
||||
- `[2026-06-13]` **arbo engine + frontend stack stood up** (ADR-0001) — irv-ml1 co-located inference engine (`ee57e69`), python-based healthcheck (slim image, no curl/wget, `bdb3312`), frontend ro-mounted from the checkout (`922e8ad`, ADR-0001 D2).
|
||||
|
||||
- `[2026-06-11]` **GLM thinking inverted at the LiteLLM gateway** (operator call): `glm-5.1` defaults thinking-OFF; `glm-5.1-reasoning` = same z.ai upstream, thinking ON. Mechanism: `litellm_params.extra_body:{thinking:{type:disabled}}` — `drop_params` strips a top-level `thinking`/`reasoning_effort` but forwards `extra_body` verbatim to z.ai. Shared-gateway change. (`95b2701`, auto-memory `reference_litellm_gateway`)
|
||||
|
||||
- `[2026-06-11]` **pi coding agent installed on nh3-dev as a GLM 5.1 harness** — `@earendil-works/pi-coding-agent` via **bun** (npm's global prefix is `/usr` → needs sudo, bun avoids it). Config `~/.pi/agent/models.json`, launcher `~/.local/bin/glm`.
|
||||
|
||||
- `[2026-06-11]` **z.ai web-tools (regin) = z.ai hosted MCP path, NOT the `/paas/v4` Tool API.** WORKS: MCP streamable-HTTP at `https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp`, `Authorization: Bearer $ZAI_API_KEY` (the **MCP** key, distinct from `Z_AI_API_KEY` the LLM key). Reference impl = Worldtree's Leif agent.
|
||||
|
||||
- `[2026-06-10]` **Mac Pro migration framed: workstation-only** (M2 Ultra ARM, racked NH3 on-subnet); sidecars stay Linux. `migration-plan.md`. (See in-flight.)
|
||||
|
||||
- `[2026-06-10]` **Worldtree deployed-config propagation is infra-ops's OWNED lane** (operator ruling). worldtree-dev pings the config delta pre-merge; infra-ops syncs `config/*.yaml` from MERGED canonical to the `/opt/worldtree*/config` bind-mounts on demo+personal. The v0.33.8 9-HOUR demo outage (a `model_roles.yaml` startup-dep that never reached the VM) is the failure mode this prevents. providers.yaml stays hand-tuned. corviduo emergency-ops = `ssh vh@10.250.50.152`, docker no-sudo. (auto-memory `reference_worldtree_deploys_cicd`, `reference_corviduo_dev_emergency_ops`)
|
||||
|
||||
- `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr-smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev — granite, later extended to qwen-vision). Mint via `/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus. (auto-memory `reference_litellm_gateway`)
|
||||
|
||||
_56 older entries archived to archival-memory.md._
|
||||
_73 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-06-16]` **litellm 500 `Router.acompletion()/aembedding() missing 'messages'/'input'` = a request missing `Content-Type: application/json`, NOT a gateway outage.** curl `-d` defaults to form-encoding → litellm can't parse the JSON body → `data` reaches the router without `messages`/`input` → 500 (should be a 400; litellm #16993). My own diagnostic calls dropped the header → I misread it as a gateway outage and needlessly bounced the gateway ~4× chasing a phantom (image/version/config were fine throughout; a malformed UI-added "Mistral Story Eval" model in the DB was a red herring I deleted). ALWAYS send `-H "Content-Type: application/json"` testing litellm; reproduce with a header'd call before declaring a litellm incident.
|
||||
|
||||
- `[2026-06-16]` **LiteLLM admin-UI playground can't test vLLM-backed models** — it auto-sends empty `tools:[]`, vLLM 400s (litellm #6228); the gateway `strip_empty_tools` hook is a PROXY hook and structurally can't reach the UI's in-process `litellm.completion()` call. Off-ramp = `tools/gateway-chat.html`. (Langfuse playground also out: its SSRF guard blocks internal-IP LLM connections, wontfix Langfuse #13097.) (auto-memory `reference_litellm_ui_playground_vllm_deadend`)
|
||||
|
||||
- `[2026-06-16]` **A gitea Actions run that "looks like it never fired" (no image, untouched build dir) usually DID fire then skipped or failed-fast.** Bit arbo TWICE: run #3 looked-never-fired but SKIPPED (change-detection `engine=0`); run #4 looked-never-fired but FAILED in ~7s (early-step). Check the run LIST + status via claude-bot (`GET /repos/{o}/{r}/actions/tasks`) BEFORE a runner-connection/dispatch theory. Gitea doesn't expose action logs via API (404 on `/runs/{id}/logs`) — read them in the UI (`/actions/runs/{id}`).
|
||||
|
||||
- `[2026-06-16]` **vLLM 0.23.0 breaks Mistral MULTIMODAL at startup** (`MistralCommonImageProcessor
|
||||
has no attribute fetch_images`, #44911 regression ~0.22.1+; same class hit Mistral-3.1/
|
||||
Devstral). The dummy-image profiling crashes. Workarounds: `--limit-mm-per-prompt
|
||||
@@ -388,18 +370,4 @@ _56 older entries archived to archival-memory.md._
|
||||
|
||||
- `[2026-06-14]` **Chatterbox-Turbo decode-knob foot-guns** (R16 v1-polish + emergent probes): the turbo length cap is `max_gen_len` (default 1000) on `t3.inference_turbo`, NOT `max_new_tokens` — and `tts_turbo.generate` does NOT forward it (wrap inference_turbo to cap). `rep_pen 2.0 / temp 0.5` BACKFIRES (degenerate 24 s run-on). Soft decode tamers cut the NVV ITSELF, not just the run-on tail (operator: "p1 trims the moaning too") — gen-time polish can't beat v1's defaults. Inline base-NVV tokens DEGENERATE (moan-cascade + gibberish), they don't modulate the surrounding words.
|
||||
|
||||
- `[2026-06-13]` **Loading an old LoRA adapter after a vocab bump fails on embedding size.** The harness config + `tokenizer.json` are now at `new_vocab_size=50279` (v4 multi-tag); the v1/v2/v3 adapters are 50277. To load v1 (the accepted artifact), set `cfg.new_vocab_size=50277` before `load_finetuned_engine_lora` (else PeftModel state_dict size mismatch). `tokenizer.json.v3bak` is the 50277 tokenizer for a clean restore.
|
||||
|
||||
- `[2026-06-13]` **Heavy from-source compile (`MAX_JOBS=128`) on the shared PROD GPU box PINS it** — load hit 187, prod vLLM restarted, killed an in-flight quant. ana-ml2 hosts live inference; never run a big build there at full parallelism. Cap `MAX_JOBS≤32`, build off-box, or cgroup-constrain.
|
||||
|
||||
- `[2026-06-13]` **`--quantization fp8` on a VL model can quantize the VISION TOWER → garbage vision** (Qwen3.5-VL on stable vLLM: gray-grid output; LM answers text fine, so it "looks" healthy). The nightly excludes the vision tower. Validate the VISION path on a quantized VLM, not just text — and pin the engine digest with the exclusion.
|
||||
|
||||
- `[2026-06-13]` **vLLM's `--gpu-memory-utilization` is checked against FREE VRAM at startup, not total** — on a shared card, growing one service before trimming a co-tenant OOMs. Trim the shrinking service FIRST, then grow. Size to the FREE budget.
|
||||
|
||||
- `[2026-06-13]` **The `vllm/vllm-openai` entrypoint is already `["vllm","serve"]`** — compose `command:` supplies the model as the first POSITIONAL arg + flags; a second `serve`/`--model X` → "unrecognized arguments". Same-class: `tee` masks the real exit code (use `>`); HF `datasets` rejects bare `wikitext` (needs `Salesforce/wikitext`).
|
||||
|
||||
- `[2026-06-13]` **Chatterbox-Turbo LoRA finetune: the repo's `setup.py` loads the WRONG tokenizer** — pulls gpt2-medium + a grapheme merge file instead of the chatterbox-turbo GPT2 tokenizer (vocab.json+merges.txt, len 50276). Fix = override + delete the grapheme `tokenizer.json`; `[vmoan]` → new_vocab_size 50277 (1-row resize), lora_r 64 / alpha 128, modules_to_save=[text_emb,text_head]. Unique-stem corpus collision needs `{index}_{stem}` IDs. (`irv-ml1:~/r16-vmoan-harness`)
|
||||
|
||||
- `[2026-06-11]` **A completion-poll `while pgrep -f <scriptname>` SELF-MATCHES its own remote shell argv** — its own `pgrep -f` always finds itself → the loop never exits. Use a match pattern ABSENT from the poll command (the python stage, or a sentinel file), not the driver's own name.
|
||||
|
||||
_58 older entries archived to archival-memory.md._
|
||||
_65 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user