vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7a59de3afa66d7b6adde85ea0b3ad1327fd5db1d
esh-pfi-infrastructure/persistent-memory.md
T
vh 7a59de3afa memory: record brokkr frozen-scorer composites (Deckard 2.176 deployed) + Deckard GDN-hybrid KV/ctx note 
char-rp-reasoning A/B closed: Deckard median composite 2.176 (0 loop/0 refuse) beats RpR-v4
3.716; Pantheon-Reasoning 1.383 (cleanest prose but 7/30 refusals, rejected). Deckard is
GDN-hybrid (16 KV layers) so its ctx scales cheaply (40K->256K ~+7G).
2026-07-08 15:22:45 -07:00

45 KiB
Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-07-08

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo Role CI status
vh/task-board MCP + web dashboard for assistant task state (port 7878) push-to-main → CI deploys (2026-04-29)
vh/vor Inquisitor UI sidecar (port 7879) push-to-main → CI deploys (2026-04-29)
vh/nevermore Twice-daily LLM-curated briefing (port 8181, replaces news-digest) push-to-main → CI deploys (2026-04-30)
vh/asset-engine Internal control plane over inference services (port 8200, LAN-direct) push-to-main → CI deploys (2026-05-12)
vh/althing Lean trusted inter-agent message bus — v2 "email model" (v2.0.0b2, 2026-07): per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API /owner/* / althing-mcp stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. per-box uv tool install (NOT CI-deploy); nh3-dev = the DEV box (editable install of ~/development/althing, gets new versions first); nh3-extdev a mesh peer (model B: althing-svc + shared /srv/althing)
vh/mead-hall Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) push-to-main → CI deploys (2026-05-16)
vh/skaldsong Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) push-to-main → CI deploys (2026-05-19)
vh/Worldtree Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. push-to-main → CI build-and-deploy (runner on ana-docker)
vh/yt-voice-clipper YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see docs/runbooks/ytvc-autodeploy.md
vh/arbo Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
model-training-forge (mtf-dev) Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward) training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

  • Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)

  • pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

  • Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.

  • Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

  • Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.

  • gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.

  • docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.

  • scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.

  • Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-training pivoted away):

  • SERVING TOPOLOGY, ana-ml2 GPU0 (co-resident), served via the qwen36-27b-aeon + ms32-24b-angel compose stacks:

    • gen (:8015) = qwen36-35b-a3b-heretic-nvfp4 — LIVE + GOOD. Off-the-shelf NVFP4 of llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only (MoE 256e/8-active = qwen3_5_moe, vision, GDN-hybrid; modelopt format → --quantization modelopt; vision bf16; 24G at /tank/aimodels/qwen36-35b-a3b-heretic-nvfp4). UNCENSORED (writes brutal fiction think-off, no system prompt). "Blathers" → needs adequate max_tokens (thinking-on looks EMPTY under ~600 tok, fine at 3k — NOT broken). General-assistant seat. Served by /opt/docker/compose/qwen36-27b-aeon/ (.env repointed: AEON_GEN_MODEL, AEON_GEN_QUANT=modelopt, AEON_GEN_GPU_MEM_UTIL=0.40; the --quantization line is now ${AEON_GEN_QUANT}-parameterized). Served-names qwen3.6-27b-aeon/-thinking (gateway gen/gen-reasoning stay transparent). util 0.37 (~35G; dropped from 0.40 on 2026-07-08 to fund char-rp 128K — the RP seats are now co-resident, ~4.4G GPU0 margin). AEON + the intermediate Pantheon-27B serves are RETIRED.
    • char-rp (:8016) = TheDrummer_Magidonia-24B-v4.3-Q6_K GGUF — LIVE + EXCELLENT. Magistral (Mistral) dark-romantasy RP tune via llama.cpp (char-rp-gguf stack, container llama-charrp). NON-thinking PROSE seat: elite literary prose, zero refusal on dark/explicit scenes, ~65 tok/s, tight POV/instruction adherence (live-tested). Replaced the broken Angel NVFP4. Alt prose model (.env swap CHARRP_MODEL): MS3.2-PaintedFantasy-v4.1-24B (more literary flair, looser POV). Both GGUFs pre-pulled at /tank/aimodels/llm/rp/.
    • char-rp-reasoning (:8018) = Qwen3.5-27B-Deckard-PKD-Heretic-Uncensored-Thinking.i1-Q5_K_M GGUF — LIVE (A/B WINNER 2026-07-08, REPLACED RpR-v4). DavidAU creative (PKD) tune on Qwen3.5-27B (container llama-charrp-reasoning). Native Qwen3.5 template → --reasoning on managed CoT SURFACES in reasoning_content, content stays clean prose, --reasoning-budget 400 caps it; DRY server-side (sampler order = dry-after-temp) tames looping. A/B: 0/30 loops + 0/30 refusals, ~57 tok/s. Runs on the base llama-swap b8840 image (Qwen3.5). Best-of-breed-per-seat (NOT the same model as char-rp).

  • 🦄 RP-SEAT UNICORN — RESOLVED + DEPLOYED (2026-07-08). char-rp = Magidonia-24B-v4.3 (Magistral prose, 65tps); char-rp-reasoning = Deckard-PKD (Qwen3.5-27B) (managed reasoning + DRY, ~57tps; A/B winner over RpR-v4 — see re-A/B note below). Both GGUF via llama.cpp (char-rp-gguf stack, ana-ml2 GPU0, ~86/97G co-resident with gen, ~11G margin). Canonical stack in repo stacks/char-rp-gguf/; gateway rewired (char-rp→:8016, char-rp-reasoning→:8018, Mistral/QwQ samplers, dropped the Qwen enable_thinking kwarg). KEY FINDINGS: (a) no single dense 24-32B is BOTH an elite non-thinking prose seat AND a clean managed-reasoning seat on llama.cpp → best-of-breed per seat; (b) Magidonia's Magistral [THINK] discipline is loose (won't close [/THINK] on substantive reasoning → prose bleeds into reasoning, content empties); Cydonia-R1's <think> is emergent (llama.cpp can't manage/cap → runaway CoT never reaches prose); QwQ's template opens <think> natively → llama.cpp manages+caps it; (c) Pantheon-Reasoning-27B stays REJECTED (re-censors in <think>; RpR-v4's abliterated-reasoning is the fix); (d) reasoning now SURFACES in reasoning_content (fixes the old AEON trace-not-surfacing gap). One-model fallback (Magidonia both, lighter reasoning) documented in the stack header/README. Requirements met: prose#1, ≥50tps#2, low-refusal#3, dense#4, GGUF-not-Ollama#5, fit-GPU0#6, thinking#7. Candidate GGUFs also on disk for A/B: Cydonia-R1-24B-v4.1, PaintedFantasy-v4.1-24B, RpR-v4 Q6_K. REASONING-SEAT RE-A/B (2026-07-08, later — RpR→Deckard): operator wanted a reasoning model that TAKES DRY (RpR forbids rep/DRY → 1/30 loop tail). Full A/B on brokkr's 30-prompt D1 suite (content-only, slop-scored): Deckard-PKD (Qwen3.5-27B) WON (0/30 loops, 0/30 refusals, clean) → NOW the char-rp-reasoning seat. RpR-v4: 0 refusals but 1/30 loop (no-DRY). Pantheon-Reasoning-27B: 7/30 explicit refusals (DeepSeek-distilled re-censor IS real across the batch, milder than feared; NOT wholesale-rejected) — kept on disk as alt. Snowdrop-v0.5-Type-S + Gembrain-31B (Gemma-4): FLOORED — llama.cpp can't do MANAGED reasoning on them (Snowdrop's ChatML template has no <think>/enable_thinking hook; Gemma-4's reasoning parser splits content wrong). GATE for a llama.cpp reasoning seat = STOCK template natively opens <think> or has enable_thinking (Qwen3.x/QwQ do; ChatML + Gemma-4 don't). INFRA: llama-swap b8840 image can't load Qwen3.6/Gemma-4 archs → use ghcr.io/ggml-org/llama.cpp:server-cuda (newer, pulled on ana-ml2) for those; Deckard (Qwen3.5) runs on b8840. brokkr frozen-scorer CONFIRMED (median composite, content-only; lower=cleaner): Deckard 2.176 (0 loop/0 refuse — DEPLOYED, big win over RpR 3.716 + its loop-tail); Pantheon-Reasoning 1.383 (cleanest prose of the field — beats even Magidonia char-rp 1.774 — BUT 7/30 explicit refusals → rejected: clean-but-refuses loses to clean-and-uncensored). MEMORY-LAYOUT note: Deckard (Qwen3.5-27B) is GDN-hybrid → only 16 KV-caching layers → KV is TINY (1.4G @ 40K vs Magidonia's 10.9G @ 128K, 40 dense layers); Deckard native ctx = 256K, so the reasoning seat's ctx scales nearly free (40K→128K ≈ +3G, →256K ≈ +7G). GPU0 layout: gen ~35G (256K/16-seq/util0.37), char-rp Magidonia ~30G (128K), char-rp-reasoning Deckard ~21G (40K), ~11.5G free. MAX CONTEXT (2026-07-08): char-rp 128K (Magidonia FULL native 131072), char-rp-reasoning 40K (QwQ native 40960, YaRN-free max), q8_0 KV cache both (near-lossless, ~2× ctx/GB; verified coherent, no Qwen gibberish). Funded by gen util 0.40→0.37 (freed ~2.9G of gen's IDLE KV headroom — gen KV usage runs 0-2%, so its 4.74×→3.66× concurrency-at-256K cut is invisible). GPU0 ~93/97G, ~4.4G margin (gen fixed-util + static KV → stable, won't OOM). 64/50 tps. kv_unified → a single conversation gets the FULL n_ctx (4 slots share the pool, not a /4 split). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE / AEON_GEN_GPU_MEM_UTIL.

  • ✅ LiteLLM gateway sampling defaults WIRED for all 4 seats (2026-07-08) — dvalin-smithy canonical, infra-ops triaged + char-rp A/B-validated. gen (+summarizer-large twin, shared :8015 deployment): temp 0.7 / top_p 0.8 / top_k 20 / presence_penalty 1.5 (non-thinking). gen-reasoning: temp 1.0 / top_p 0.95 / top_k 20 / presence_penalty 1.5 (Qwen general-thinking profile — the old temp 0.6 was the CODING sub-profile by mistake; verbose ~9k-char CoT → callers need big max_tokens). char-rp: temp 1.1 / top_p 0.95 / min_p 0.10 / top_k 0, NO rep (A/B: min_p 0.10 richened imagery vs 0.03; rep 1.05 REJECTED — injected a markdown title in a grief scene, matches Drummer-no-rep). char-rp-reasoning: temp 1.0 / top_p 0.95 / top_k 40 / min_p 0.02, no rep/DRY. Canonical doc docs/pfi/model-sampler-defaults.md; dvalin derivation dvalin-smithy/hoard-drafts/pfi-gateway-sampler-defaults-20260708.md.

  • NVFP4/GGUF quant tooling (proven, on ana-ml2): /tank/aimodels/quant-work/ — nvfp4_quant.py (llm-compressor, MoE-aware load_quantizable_moe, Mistral-tuned ignore) + .venv; qwen35_vision_nvfp4.py (llm-compressor + qwen ignore + trust_remote_code + a torchvision-gate bypass). ModelOpt path: /home/lkraven/nvfp4-quant/.venv-mopt (modelopt 0.44) + mopt_w4a16.py (= mtq.quantize + export_hf_checkpoint → modelopt format w/ hf_quant_config.json). Downloads via a detached vLLM container running huggingface_hub.snapshot_download to /tank/aimodels/<name>-{bf16,nvfp4}. Gotchas: llm-compressor save_compressed DROPS tokenizer/chat/preprocessor files → stage from the bf16 source; the qwen ignore keeps re:.*visual.* + re:.*linear_attn.*(GDN) + lm_head + embeds + mlp.gate(router) bf16.

  • 🪦 T1 / home-trained LoRA — DORMANT (pivoted to off-the-shelf). DPO came back slop → operator killed home-training (2026-07-08). ONLY this project dormant; LEAVE arbo's LoRA-worker (irv-ml1 :8203), ALL /tank models, and every other project ("ritual seppuku" otherwise). Revival map: adapters data/spike/{sft,dpo}_adapter, harness canonical v0.0.44, full saga in Recent decisions + mtf-dev auto-memory; aeon-trainer image left (28G). mtf-dev stood down. (Detailed 2026-07-07 hot-swap findings preserved below + in Recent decisions.)

Earlier (2026-07-07):

  • T1 SFT COMPLETE (AEON-27B E-RP LoRA) — adapter banked, but BLOCKED on hot-swap serving. SFT ran clean on ana-ml2 GPU0 (all-TRL torch-GDN, batch1/accum8/GC-off, 1 epoch, ~9h @ 0.25 s/s steady, final loss 2.445); adapter at ana-ml2:~/model-training-forge/data/spike/sft_adapter (124MB, r16 α32, targets self_attn.q/k/v/o_proj + linear_attn.in_proj_qkv/out_proj, NO in_proj_z). DPO leg deferred (needs mergekit resolved + on-policy rejected regen). THE WALL: the adapter is INERT on the vLLM NVFP4 hot-load — loads HTTP 200 but zero deltas reach inference (vLLM 0.24.0 qwen3_5 LoRA-APPLICATION silent no-op; upstream #47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4 AND FP8 both inert) AND adapter-format-agnostic (separate-key sft_adapter_zc loads clean via vLLM's allowlist but inert; the fused-key rekey rejected outright). The #47640 guard-patch overlay did NOT fix it. HARD REQUIREMENT (operator): hot-swappable or ABANDON — merge is NOT a solution now (static; reserved for a later RP-quant track only).

  • HOT-SWAP — SGLang zc smoke + 2-line patch RAN (2026-07-07). PARTIAL PROGRESS: patch cleared startup validation + self-attn LoRA loads end-to-end, but GDN in_proj hit 2 MORE name-routing gaps → NOT a one-line fix. MERGE RECOMMENDED (operator's call, pending). Ran sft_adapter_zc on SGLang 0.5.13 bf16 (--lora-target-modules all --max-lora-rank 16 --lora-paths t1=/adapters/sft_adapter_zc, GPU0, gen+rp stopped for the smoke then restored). Base weights loaded clean; self-attn q/k/v/o normalized + accepted fine. Died at lora_manager.init_lora_shapes with a NAME-VALIDATION ValueError (NOT the feared in_proj_qkvz shape-assert): adapter's in_proj_qkv+in_proj_z "not included in --lora-target-modules [...in_proj_qkvz...]". ROOT CAUSE (pinned): srt/lora/utils.py:235 get_normalized_target_modules uses a HARDCODED params_mapping dict that maps q/k/v_proj→qkv_proj (why self-attn works) but OMITS in_proj_qkv→in_proj_qkvz + in_proj_z→in_proj_qkvz; it does NOT consult the model's packed_modules_mapping (which DOES declare "in_proj_qkvz":["in_proj_qkv","in_proj_z"] at models/qwen3_5.py:1010). Everything else is ALREADY in place: get_hidden_dim("in_proj_qkvz") IS implemented in the model class (:1040, → (hidden_size, key_dim*2+value_dim*2); the "utils NotImplementedError" was only the DEFAULT); get_stacked_multiply("in_proj_qkvz")=4 in the utils default dict; base-weight stacked_params_mapping fuses in_proj_qkv→shards(0,1,2),in_proj_z→shard 3. PATCH APPLIED + RE-SMOKED (bind-mounted a patched utils.py overlay into the container): the 2-line add cleared init_lora_shapes; server came up "fired up and ready", LoRA kernels compiled (K=16384 in_proj_qkvz expand + K=6144 in_proj_z shrink — B-side math right per mtf-dev), self-attn loaded end-to-end ("loaded weights for [o_proj,out_proj, qkv_proj]"). BUT GDN in_proj died at TWO MORE name-resolution points (name-routing, NOT shape/intractable): (1) adapter-load SKIPPED 192 in_proj_qkv/in_proj_z weights ("did not match any target module [...in_proj_qkvz...]"); (2) inference buffer-fill utils.py:300 get_target_module_name RAISED — SUBSTRING match, in_proj_qkvz ⊄ ...in_proj_qkv. and qkv_proj ⊄ it → "Cannot find target module name for ...in_proj_qkv.lora_A.weight" → HTTP 500 + scheduler killed. So SGLang's GDN LoRA name-routing is incomplete in ≥3 code paths (fixed 1/3). FINISH LINE (if ever revived): teach get_target_module_name + the load-consolidation the in_proj_qkv/in_proj_z→in_proj_qkvz map (mirror qkv_proj, which works E2E), THEN mtf-dev's buffer shard-placement (0,1,2)+3 question goes live again (still UNPROVEN, deepest step). Per operator's "one targeted fix else merge" rule → this is multi-point, NOT one fix → MERGE recommended. mtf-dev replied + loop closed (thread 01KWZDZFYN…). (SGLang GENERIC image still can't LOAD our NVFP4 checkpoint — bf16-only smoke.) Adapter variants: sft_adapter (separate), sft_adapter_zc (separate + zero in_proj_z, the canonical one — THIS smoke), sft_adapter_fused (fused — wrong), plus sft_adapter_attnonly. All at /home/lkraven/model-training-forge/data/spike/.

  • MERGE FALLBACK — primed (~2-3h) if hot-swap is fully abandoned. peft merge_and_unload (~20min, LOW wall — separate keys merge directly, no rekey) → sanity greedy-diff (finally reveals if the ~10h training is any good — we've NEVER seen the LoRA apply) → ModelOpt NVFP4 re-quant of the merged bf16 (~1-2h, MEDIUM-HIGH wall = the GDN-hybrid requant + calibration + verify GDN modules quantize) → serve as plain base (no --enable-lora). Static, NOT hot-swappable.

  • comfy-dev in-arbo LoRA-training worker — DONE (all phases, this session). Built/deployed the worker (services/lora-training-worker/, irv-ml1 0.0.0.0:8203, runs as llmuser, fixed-invocation sd-scripts, systemd, elway deploy playbook; commits 888ba6a→b617a8b). Phase 1 first real train SUCCEEDED (Sindra fast/lean). Phase 2 publish-step (published_lora_name + optional train_id field). Phase 2.5 GC contract (shape A: GET /published-loras + POST /gc/published {train_ids}) CONFIRMED — worker impl DEFERRED to after the T1 hot-swap work. auto-memory reference_lora_training_worker.

  • Serving state (SUPERSEDED — see top; AEON retired 2026-07-08): during the T1 work, AEON gen (:8015) + char-rp (:8016) NVFP4 were HEALTHY, stopped/restored for hot-swap smokes via docker start vllm-aeon-gen vllm-aeon-rp. Now replaced: gen = 35B-A3B heretic, char-rp = (broken Angel, being replaced).

  • worldtree-dev vh/pypi read token delivered (bifrost==1.1.0 pin, #349); dedicated read:package token at nh3-dev:~/.config/gitea-tokens/vh-pypi-read.txt. The over-scoped write:package token the operator pasted was revoked + verified dead.

  • Backups — STILL OPEN: rotate the 5 disclosed rest-server creds (operator, offline); confirm esh-vm-db's resticprofile includes DB dumps. docs/runbooks/backups.md.

  • Standing / parked: phantom qwen3.6-35b-a3b gateway cleanup (400s in /v1/models); ana-docker docker-daemon log-cap; inert mood.decay_rate/stale_hours keys in deployed WT config; Mac Pro migration (hw-gated); R22 gateway-key at /home/lkraven/.r22-gateway-key (mode 600, paid GLM, do NOT delete); Deckard staged on ana-ml2 as T1's writing benchmark; /books transient NFS mount on nh3-dev; dvalin sampler re-tune pass for char-rp / char-rp-reasoning once they accumulate live Worldtree/ST character-role traffic (dvalin offered 2026-07-08 to do a follow-up after more live sessions; the current paper defaults + 2-prompt A/B are the starting point — real long-session usage is where repetition/drift surfaces; re-open thread 01KX1DS6… to take them up on it).

  • Worldtree config-propagation (reference): demo+personal bind-mount config from /opt/worldtree{,-personal}/config (infra-ops-deployable, byte-identical from canonical); reload via docker restart <container>, NEVER compose up (stale-:latest footgun). Role registry loaded ONCE + CACHED at startup → a bind-mount model_roles.yaml change needs a restart; pre-stage BEFORE the activating deploy's restart. config REMOVALS are NOT backward-compatible with the running image.

Recent decisions

  • [2026-07-08] OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training. Final topology: gen = llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only (LIVE, modelopt, vision, util 0.40), char-rp = an RP unicorn to be found on fresh context (see Current state). Intermediate steps ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path); Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway sampling-defaults wiring still PENDING.

  • [2026-07-08] DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call). Root cause: DpoConfig had NO epochs field + _dpo_config_kwargs didn't pass num_train_epochs → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs

    • _sft_config_kwargs). Objective SATURATED by epoch 0.27 (loss0, grad0, acc 1.0, margins27 flat — the off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run. FIX (3 edits to deployed harness, mtf-dev to canonicalize): DpoConfig.epochs: float = 1 (mirrors SftConfig, float for fractions); _dpo_config_kwargs now passes "num_train_epochs": cfg.epochs; recipe dpo.epochs: 0.3. GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min train = ~45min). DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step; train_loss 1.4e-5 @ epoch 0.301). Fresh data/spike/dpo_adapter/adapter_model.safetensors (123MB) + checkpoint-45 banked. 3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0): ALL THREE DISTINCT (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. DPO 0.3ep is COHERENT, fluent, NOT degenerate (early-stop avoided over-optimization) but the quality delta on a neutral literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift, arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval + the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).

  • [2026-07-08] T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py. Blocker resolved: (1) mtf-dev's v0.0.42 stub _stub_missing_optional_integrations (last-resort sys.meta_path finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied VERBATIM to deployed src/model_training_forge/train/backend.py after _unsloth_available() + call-site before from trl import DPOTrainer); (2) my cosmetic warnings_issued shim (trl-0.24 DPOTrainer.init:405 does model.warnings_issued["estimate_tokens"]=True for warning-suppression; custom Qwen3_5 class under transformers 5.5.0 lacks the attr → if not hasattr(model,"warnings_issued"): model.warnings_issued={} before the DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED un-git'd copy only → mtf-dev must canonicalize the warnings_issued shim into their repo (told them). DPO confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93% util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit. gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.

  • [2026-07-08] T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe. Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. PROVEN LAUNCH RECIPE (replicates the SFT container aeon-t1-sft exactly, only --stage sft→dpo): sudo docker run -d --name aeon-t1-dpo --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml --stage dpo. CRITICAL: --entrypoint python3 is REQUIRED — aeon-trainer's default entrypoint is ["vllm","serve"] (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash. Dataset verified (pairs_dataset=train.flat.json=1196 pairs). THE BLOCK: from trl import DPOTrainer (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→ immutables→mergekit-0.1.4↔pydantic-2.13 HARD incompat (needs pydantic==2.10.6)→llm_blender→dataclasses_json→ llm_blender-0.0.2↔transformers-5.5.0 HARD incompat (TRANSFORMERS_CACHE removed, needs source patch)→weave→ (more). NONE used by our pair-based DPO. pip install mergekit w/deps is UNSAFE (downgrades accelerate 1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0) but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image. Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX, expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.

  • [2026-07-07] SGLang zc hot-swap: 2-line patch RAN → PARTIAL. Cleared startup + self-attn E2E, GDN in_proj hit 2 MORE name-routing gaps → MERGE recommended. Root cause was a NAME-routing gap (not the feared shape-assert): lora/utils.py:235 hardcoded params_mapping lacked in_proj_qkv/in_proj_z → in_proj_qkvz. Patched those 2 lines (overlay bind-mount) → init_lora_shapes passed, server up, self-attn loaded end-to-end. But the SAME missing GDN mapping bit 2 MORE resolvers: adapter-load skipped 192 in_proj weights, and inference utils.py:300 get_target_module_name (substring match) RAISED → 500 + scheduler died. So GDN LoRA name-routing is incomplete in ≥3 SGLang paths (fixed 1). NOT shape-intractable (self-attn proves the separate-key→fused-buffer machinery works), but NOT "one targeted fix" either → per operator's LAST-shot rule → MERGE. mtf-dev looped + closed (thread 01KWZDZFYN…); B/A shard-math never reached. Supersedes the "SGLang wants a THIRD format" framing — no new adapter format needed.

  • [2026-07-07] T1 (retargeted qwopus-122B→AEON-27B) SFT COMPLETE on ana-ml2 — all-TRL torch-GDN, batch1/accum8/GC-off, 1 epoch ~9h @ 0.25 s/s, loss 2.445, adapter banked. Supersedes the 2026-07-05 cloud-vs-smoke venue decision: AEON-27B's 52G bf16 fits one Blackwell → ran on-prem in ~9h (no cloud, no full-fleet outage). GC is NOT the throughput lever (batch1-no-GC == batch4+GC at 0.25; dvalin's 0.565 was warmup-noise from a 30-sample smoke); ~0.25 s/s is the torch-GDN floor (only FLA beats it, sm_120- blocked #913). reference_t1_cloud_train_plan.

  • [2026-07-07] T1 hot-swap on vLLM NVFP4 is DEAD — vLLM 0.24.0 qwen3_5 LoRA-application silent no-op (#47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4+FP8) AND adapter-format-agnostic (separate zc loads-but-inert; fused rejected); #47640 guard-patch overlay didn't fix. Root-caused by a 3-peer dwarf panel (eitri/regin/dvalin); their vLLM issue refs verified real via GitHub API. Engine mirror-image: vLLM wants SEPARATE keys, SGLang wants FUSED.

  • [2026-07-07] HARD REQUIREMENT restated: hot-swappable LoRA or ABANDON. Merge is NOT a solution for this requirement (static; reserved for a later RP-quant track). SGLang is the last hot-swap shot; if it can't apply, park the adapter + file upstream repros. reference_lora_training_worker (adjacent).

  • [2026-07-07] comfy-dev in-arbo LoRA-training worker DONE — built/deployed/validated Phases 1+2 (first Sindra train succeeded, publish-step + train_id), Phase 2.5 GC contract (shape A) confirmed + impl deferred. services/lora-training-worker/, commits 888ba6a→b617a8b. reference_lora_training_worker.

  • [2026-07-06] infra-ops NOPASSWD sudo confirmed on irv-ml1 + ana-ml2 (the arbo memory's "NOT infra-ops on irv-ml1" was stale) — enabled the worker deploy + every ana-ml2 hot-swap smoke.

  • [2026-07-05] T1 training venue: CLOUD recommended; operator chose smoke-first on ana-ml2. On-prem ruled out (ana-ml2 full — both 96G GPUs ~93G used): keep-serving = NVMe offload ~6-8 DAYS; full ana-ml2 shutdown = CPU offload ~1 DAY but a whole-fleet outage. Cloud Vast.ai 8×80GB (no offload → ~3-6h, ~$60-500, zero fleet impact) is the clean alt (mtf-dev + infra-ops both rec; Vast for its no-content-AUP marketplace + likely-existing VastBlue account). Operator's next step = the ana-ml2 CPU-offload SMOKE (~60 steps) to get real samples/sec before the full-outage-vs-cloud call. HF base verified public (zero-auth pull). Runbook + gotchas in reference_t1_cloud_train_plan.

  • [2026-07-05] glm-5.2 canonical limits recorded (probed live vs z.ai): 1,048,576 (1M) input context / 131,072 (128K) max output; NO gateway-side cap (pure passthrough → z.ai's limits are effective). Written to the config comment (commit 624a07e) + reference_litellm_gateway.

  • [2026-07-04] character-rp: gateway-forwarding proven + role pre-staged + #344 shipped. Empirically confirmed per-request extra_body (top_k/repetition_penalty) forwards through the gen-reasoning LiteLLM alias to vLLM + standard params override the alias defaults — no gateway cap needed (I over-built a dedicated alias, operator corrected, reverted with zero fleet impact). Pre-staged the character-rp role into demo+personal bind-mount model_roles.yaml (byte-verified on b18; caught the cached-registry ordering). worldtree-dev shipped #344 (v1.0.0b19) for the durable-agent override-drop. spend_logs spot-check queued (ratatoskr's timestamp ping).

  • [2026-07-04] althing v2 herald+receiver formalized as systemd on nh3-dev. althing-herald.service (Restart=always, Environment=PATH incl ~/.cargo/bin — the pane-dispatch fix) + althing-receiver.service (v2 → pillar-3 /owner/* live); stale forseti unit removed; both on v2.0.0b2, canonicalized by forseti. reference_nh3_dev_althing_herald.

  • [2026-07-04] LitBench-RM respun (irv-ml1 A6000, comfyui displaced) for T1's reward ensemble; operator sole comfyui consumer, holding image-gen until LitBench done. reference_litbench_rm_irv_ml1.

  • [2026-07-03] ratatoskr-dev DEMO Heimdall key provisioned (R30 φ0). Minted a tier-user key on the demo via POST /admin/keys (bootstrap admin key), mirroring their personal base consumer (no character-binding); base-agent affect reads work ungated. reference_worldtree_demo_key_mint.

  • [2026-07-02] mtf-dev granite harness-spike ran GREEN — MECHANICAL only, efficacy DEFERRED to the T1 run. Trainer TRL SFT→DPO→eval seam proven end-to-end on a synthetic fixture (not the E-RP corpus); operator DECIDED no intermediate real-efficacy granite spike (uninterpretable proxy — arch gap + abliteration axis). reference_gen_qwopus_122b.

  • [2026-07-01] Worldtree #332 embed-recall diagnosed + scoped-log view/tunnel provisioned + fix verified (15×→1.01× re-embed). reference_wt_gateway_scoped_log_view.

  • [2026-07-01] qwopus native MTP speculative-decode tested on gen → NOT kept (+12% single-stream, −15–20% aggregate at concurrency, silently drops min_p/logit_bias). Banked for T1. reference_gen_qwopus_122b.

  • [2026-07-01] Deckard trial → reverted to qwopus (gen) (won writing "in every way" but ~36 vs ~90 tok/s; spec-decode rescue ruled out). git b63c48b→681eb70. Deckard kept staged as T1's writing benchmark.

  • [2026-06-25] althing re-architected to the lean multi-machine bus; nh3-extdev stood up as a MODEL B mesh peer (dedicated althing-svc + group-shared /srv/althing). reference_nh3_extdev_althing_mesh.

  • [2026-06-23] zellij native web client piloted on nh3-dev (zellij-web.service :8443) alongside ttyd. reference_zellij_web_seat.

  • [2026-06-22] Worldtree persona-render config arc (#314/#322/#317) pre-synced + deployed green on demo+personal — #317 a boot-blocking config REMOVAL. reference_corviduo_dev_emergency_ops.

  • [2026-06-20] R22 (brokkr/dwarves) stood down to gateway-only; full-access R22 key minted; Phase B CANCELLED (Worldtree model-agnostic → no deploy path). Key at /home/lkraven/.r22-gateway-key (persistent mode-600, carries paid GLM, don't delete). MUT = free qwen3.5-122-a10b (gen). Operator steer: R22 research is gated on a pragmatic/deployable outcome, not advancing-the-art.

  • [2026-06-20] claude-bot issue-scope token minted for worldtree-dev self-serve (id 16, write:repository+write:issue); old token revoked. Advances the credential-migration directive.

  • [2026-06-20] rest-server-ana recovered + backup prevention shipped + worldtree-dev admin keys provisioned (demo d113207c / personal f4f75adb). Cred rotation (5 rest-server pw) BELAYED.

  • [2026-06-20] claude-bot → ADMIN on vh/Worldtree (operator-authorized) — self-serves WT deploys/tokens henceforth.

  • [2026-06-14] STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials. (auto-memory project_migrate_infra_access_to_claude_credentials)

118 older entries archived to archival-memory.md.

Tried and abandoned

  • [2026-07-08] Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE. llm-compressor W4A4 NVFP4 (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY (temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models. Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going GGUF (llama.cpp) to sidestep the whole NVFP4-quant surface.

  • [2026-07-08] Mistral3 + vLLM tokenizer/vision traps (serve MS3.2-24b, vLLM 0.24). (a) HF tokenizer.json for Mistral = GARBAGE output — the card's "use the official Mistral tokenizer" warning is REAL; must use the tekken.json/mistral tokenizer. (b) BUT --tokenizer-mode mistral + vision CRASHES (Failed to apply PixtralProcessor on {'text': '[IMG]'}; and with tekken.json present in auto mode, CachedMistralCommonBackend has no attribute is_fast). So it's mistral-tokenizer OR vision, not both on this vLLM. Text-only + mistral tokenizer serves clean (--limit-mm-per-prompt '{"image": 0}'). GGUF/llama.cpp avoids all of this (native mistral tokenizer + vision).

  • [2026-07-08] Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base. The base (llmfan46 heretic) writes freely (thinking-off), but Gryphe distilled the reasoning traces from DeepSeek 3.2 (safety-aligned) onto every turn (preserve_thinking:true) → the model reasons ITSELF into refusals in the <think> phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified). Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw abliterated base is cleaner — this is WHY the pivot went to the llmfan46 heretic base for gen.

  • [2026-07-08] Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance. MTP is a separate bf16 head (mtp.*, in model-auxiliary.safetensors, 15 tensors); AEON preserved it by INJECTING the bf16 head into the quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors main + injected bf16 mtp + text_config.mtp_num_hidden_layers=1 → vLLM detected the MTP but SKIPPED the bf16 self_attn weights → 0/192 draft tokens accepted. The bf16 MTP head only loads on the MODELOPT main-model format (like AEON), not compressed-tensors. (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.)

  • [2026-07-07] vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639). Adapter loads HTTP 200 but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate zc adapter — correct per vLLM's check_unexpected_modules allowlist — loads clean but inert; the fused-key rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of expand_packed_lora — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no version-bump helps. Merge bakes deltas in (bypasses this) but is static.

  • [2026-07-07] SGLang generic image can't LOAD our NVFP4 AEON — ModelOptModelLoader weight-shape/ packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated qwen36-27b-nvfp4 dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was quant-loader-specific).

  • [2026-07-07] SGLang --lora-target-modules CLI enum REJECTS the GDN names its own resolver asks for (invalid choice: 'in_proj_qkv'); 'all' resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants its OWN packed layout (base r16 + get_stacked_multiply=3, NOT a pre-fused rank-48 qkv → the [48]-vs-[144] shape assert). A THIRD adapter format; version-exact source needed (:latest=0.5.13, NOT main).

  • [2026-07-07] Engine invocation footguns cost several wasted serve-bounces this session — docker run --rm ate crash logs; duplicated serve (vLLM image entrypoint is already ["vllm","serve"]); --max-lora-rank 48 invalid (choices 1/8/16/32/64… → use 64); parens in echo inside ssh host -c "…" break the remote shell. LESSON: verify engine launch flags (--help, GPU-free) + never --rm a container whose crash logs you need, BEFORE bouncing a production serve.

  • [2026-07-04] LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge → my deliberately-invalid top_k=-5 forwarding-probe bled into a param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT caching (none configured), NOT a config change. Never fire invalid/distinctive sampler values at a SHARED gateway alias with live consumers — use a throwaway alias, or a docker restart litellm flushes residual carryover. feedback_litellm_shared_param_mutation.

  • [2026-07-04] A systemd --user daemon that shells out to ~/.cargo/bin/~/.local/bin tools needs an explicit Environment=PATH — the minimal --user default silently drops them. The althing herald lost zellij → silent pane-miss for ALL config-backed TUI/pane agents; CC + FIFO routes were unaffected, so it was invisible from a CC session. reference_nh3_dev_althing_herald.

  • [2026-07-04] On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS (1-GPU + NVMe ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the 566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. reference_t1_cloud_train_plan.

  • [2026-07-01] A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" is usually the pull-only-vs-build RACE, not registry-auth. deploy-personal.yml is PULL-ONLY but fires on the staging/vX tag simultaneously with deploy.yml's build → pulls before the push finishes. FIX: re-run once built, or gate on workflow_run: completed.

  • [2026-07-01] MTP/spec-decode on a SHARED serving model helps single-stream but HURTS moderate-concurrency aggregate + silently ignores min_p/logit_bias (qwopus gen: N=1 +12%, N=4 −20%). Reserve for dedicated/interactive deployments.

  • [2026-07-02] irv-ml1 /worktank ROOT is root-owned — lkraven can't write there (irv-ml1 sudo needs a password) → stage model pulls to /home. PIN THE A6000 BY UUID for training (native-CUDA ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). CUDA_VISIBLE_DEVICES=GPU-<uuid>.

  • [2026-06-25] althing "unreachable: " can MASK an app-level 500. Raw network was clean; root cause = receiver DB agents-table not synced with the config roster → delivery 500'd "unknown to: ", MAPPED to "unreachable". Diagnose: raw curl to :8087 + connect-probe ⇒ NOT network. Fixed in althing v0.17.1. reference_nh3_extdev_althing_mesh.

  • [2026-06-20] rest-server .htpasswd: permission denied = the ana-nas NFS mount FAILED (ghost file on the local mount point), NOT a decommission. mnt-backup.mount stuck failed (fstab bare defaults) → rest-server serves an empty local dir. Recovery in disaster-recovery.md.

  • [2026-06-20] The DEFAULT ssh ana-docker is lkraven (no NOPASSWD) — but ssh infra-ops@ana-docker HAS NOPASSWD root. A sudo cp as lkraven silently failed → nearly punted the rest-server recovery. Reach for infra-ops@ana-docker for sudo ops.

98 older entries archived to archival-memory.md.

Reference in New Issue View Git Blame Copy Permalink