memory: archive 15 pre-2026-06-16 entries to archival-memory.md
9 Recent decisions + 6 Tried-and-abandoned (dates [2026-06-14]/[2026-06-15]) moved non-destructively to archival-memory.md, each stamped _Archived 2026-06-19._. Kept the active [2026-06-14] 'migrate ALL infra access to Claude-specific credentials' standing directive. Back-ref counts: Recent decisions 79->88, Tried-and-abandoned 70->76. persistent-memory.md 397->363 lines.
This commit is contained in:
@@ -450,6 +450,37 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
||||
- `[2026-06-14]` **Arbo deploy pipeline fixed, hardened, and version-controlled.** Prod rebuilt v0.11.1 → **v0.11.6** backend; the webhook machinery (`arbo-deploy.sh` + `arbo-webhook.py`, :9009 HMAC listener) is now repo-tracked at `stacks/arbo/` (was host-only = recoverability foot-gun). Deploy reaches gitea via the INTERNAL route (`10.250.50.70:222`) and restarts the engine ONLY on `catalog/` changes (graphs/frontend per-request; warn on `src/`|`Dockerfile` only — pyproject/uv.lock churn every commit). Operator kept arbo stack ownership in **eshpfi** (not migrated to comfy-dev's repo). Secret + `.env` stay host-only. Tracking: `6d66bc2`, `6e58e57`, `stacks/arbo/README` Q5.
|
||||
_Archived 2026-06-18._
|
||||
|
||||
- `[2026-06-15]` **qwen36 thinking-split at the gateway** (operator) — base
|
||||
`qwen3.6-35b-a3b` = thinking-OFF default (the vision judge needs it), `…-a3b-thinking`
|
||||
= opt-in reasoning. One hybrid checkpoint + `--reasoning-parser qwen3`; mirrors the
|
||||
glm-5.1 split. The judge's enable_thinking=false is what makes the NVFP4 swap
|
||||
transparent. (`6de0844`)
|
||||
_Archived 2026-06-19._
|
||||
|
||||
- `[2026-06-15]` **arbo engine rebuilt v0.11.6 → v0.11.22 from main; catalog re-applied (v0.11.23) — curated `/workflows` footer live.** src/ schema changes (Model/Lora.summary, Workflow.blurb/keywords + the /workflows endpoint) need a MANUAL image rebuild (the webhook warns-only on `src/`/`Dockerfile`). Built from `/worktank/arbo/repo`, bumped `ARBO_IMAGE` in the host .env, recreated + verified healthy (auto-rollback armed, unneeded). comfy-dev's v0.11.20 catalog push had first crash-looped prod (old baked schema `extra=forbid` vs new git-mounted catalog fields → ValidationError); they reverted the catalog, so rebuild-from-main loaded clean, then they re-applied (v0.11.23). comfyui is a SEPARATE compose project — the rebuild didn't touch it.
|
||||
_Archived 2026-06-19._
|
||||
|
||||
- `[2026-06-15]` **althing-core refreshed to v0.14.1 box-wide on nh3-dev** (forseti + heid both routed it; central-sequenced per DevOps-delegation since `uv tool install --force` swaps the shared binary all handles use). Fixes the monitor non-truncating-lock bug so `stop-monitor`/`--stop` reliably recovers orphans. The editable install (`uv tool install --force --editable .`) covers every nh3-dev handle; `--version` trued up 0.14.0→0.14.1; verified `--stop` works against a real orphan.
|
||||
_Archived 2026-06-19._
|
||||
|
||||
- `[2026-06-15]` **ratatoskr Tier-3 affect live smoke GREEN** (consumer-confirmed). Brokered/minted consumer_id="ratatoskr" Heimdall HS256 key on personal Worldtree (the key IS BOTH the create-session Bearer AND ratatoskr's JWT verifier secret — plain utf-8 string, NOT base64/raw); injected ENV-ONLY to `~/.config/ratatoskr/provider.env` (mode 600) on nh3-dev; added `10.100.10.50:8390` to personal's `BIFROST_CLIENT_ALLOWED_HOSTS`; drove define→session (handshake 200)→turn→affect.emit (200, snapshot persisted). Worldtree signs Bifrost with the caller's raw Bearer verbatim. Drive commands + claims contract in althing thread `01KV4ZYSE3…`.
|
||||
_Archived 2026-06-19._
|
||||
|
||||
- `[2026-06-15]` **Dense Qwen3-VL-32B-NVFP4 A/B'd as arbo's anatomy hero-judge vs the qwen3.6-35b-a3b MoE → dense LOST, pilot torn down, GPU 0 reclaimed.** Dense NVFP4 loads clean on vLLM where the MoE NVFP4 doesn't (no expert-scale keys); ~44 tok/s on Blackwell GPU 0; vision tower stays high-precision. But comfy-dev's ground-truth A/B: MoE wins (faster + dual-use as general LLM + slight fine-detail edge), and the subtle-proportion tier is below ALL VLs. **Don't re-propose a dense-VL judge swap.** (auto-memory `project-qwen3vl-dense-pilot`)
|
||||
_Archived 2026-06-19._
|
||||
|
||||
- `[2026-06-15]` **MastMed prototype containerized + exposed publicly via cloudflared.** Self-contained static SPA (bun→caddy multi-stage build), basic-auth gated (operator's gating call), ana-docker:8089, cloudflared quick tunnel. See in-flight for the ephemeral-URL caveat + named-tunnel upgrade path.
|
||||
_Archived 2026-06-19._
|
||||
|
||||
- `[2026-06-15]` **infra-ops bootstrapped onto corviduo-dev** (operator-directed) — infra-ops now has SSH + NOPASSWD sudo there, OVERRIDING the fleet bootstrap script's deliberate partner-box exclusion (explicit host arg). corviduo config edits no longer need `vh` + a sudo password. (auto-memory `reference_corviduo_dev_emergency_ops`)
|
||||
_Archived 2026-06-19._
|
||||
|
||||
- `[2026-06-15]` **R18 clip+caption placement = ana-ml2** (operator: "ana-docker… ana-ml2 unless irv-ml1 has room"; irv-ml1 lacked free VRAM for the 30B-A3B captioner). Models staged + stub deployed; see in-flight + the hard real-voice gate.
|
||||
_Archived 2026-06-19._
|
||||
|
||||
- `[2026-06-14]` **LiteLLM infra-ops admin key provisioned** (operator) — resolves the LiteLLM half of the credential-migration directive; use it for ALL gateway ops (NOT `sk-corvid`). Value at `~/.config/litellm/infra-ops-key` (mode 600); gateway reachable directly from nh3-dev at 10.250.50.70:4000. (auto-memory `reference_litellm_infra_ops_key`)
|
||||
_Archived 2026-06-19._
|
||||
|
||||
## Tried and abandoned (archived)
|
||||
|
||||
- `[2026-04-30]` task-board workflow with
|
||||
@@ -907,3 +938,21 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
||||
|
||||
- `[2026-06-14]` **Chatterbox-Turbo decode-knob foot-guns** (R16 v1-polish + emergent probes): the turbo length cap is `max_gen_len` (default 1000) on `t3.inference_turbo`, NOT `max_new_tokens` — and `tts_turbo.generate` does NOT forward it (wrap inference_turbo to cap). `rep_pen 2.0 / temp 0.5` BACKFIRES (degenerate 24 s run-on). Soft decode tamers cut the NVV ITSELF, not just the run-on tail (operator: "p1 trims the moaning too") — gen-time polish can't beat v1's defaults. Inline base-NVV tokens DEGENERATE (moan-cascade + gibberish), they don't modulate the surrounding words.
|
||||
_Archived 2026-06-18._
|
||||
|
||||
- `[2026-06-15]` **A comfyui restart-loop with `exit 0` + `OOMKilled=false` is NOT OOM — suspect a native SEGFAULT.** `docker inspect` shows only the LAST exit code; a GPU OOM is a catchable `RuntimeError: CUDA out of memory`, NEVER `Fatal Python error: Segmentation fault`. Diagnostic: `RestartCount>0` = same container restarted in-place → the docker json-log ACCUMULATES across all cycles, so grep the full `docker logs` for the crash signature + the lines right before each restart banner. comfyui's was at VAE decode (sampling completes, then segfault loading `AutoencodingEngine`) — gen-triggered but native, not VRAM. Suspects logged: torch/cu130 mismatch, an injected `aimdo` cuda-funchooks lib, the VAE.
|
||||
_Archived 2026-06-19._
|
||||
|
||||
- `[2026-06-15]` **NEVER blanket-kill `althing-light-monitor` by PID** — they're per-handle and MANY agents run their own on the shared nh3-dev box (heid / ratatoskr-dev / worldtree-dev / bifrost-dev each have one). A `kill $(pgrep -f light-monitor)` downs four peers' monitors (near-miss caught by checking `/proc/<pid>/environ` `ALTHING_HANDLE` + cwd first). Recover your OWN via `althing-cli stop-monitor` (v0.14.1+); a raw `&`-backgrounded monitor is untracked (no harness fire-notification) — always arm via `run_in_background`. (extends the pkill-self-match lesson below)
|
||||
_Archived 2026-06-19._
|
||||
|
||||
- `[2026-06-15]` **Manual `docker compose up` on corviduo Worldtree DOWNGRADES to a stale local `:latest` → crash-loop (config/image mismatch).** Compose `image:` is `…/worldtree:latest`, but local `:latest` lagged ~2wk (`21c90d5eeb2d`); CI/CD deploys NEWER SHA images, and a real pull 401s (daemon has no gitea-registry creds — that's CI/CD's). `--pull never` then used the stale latest → personal crash-looped on `ConfigurationError: echo.default_model missing` → **~2-3 min personal outage**. RECOVER: retag `:latest` to the image a HEALTHY SIBLING runs (`docker inspect worldtree-worldtree-api-1 --format '{{.Image}}'` → `docker tag …`), then `up -d --pull never`; verify `Application startup complete`. Don't manually recreate Worldtree on corviduo without pinning the correct image — image+config are CI/CD-coupled. (auto-memory `reference_corviduo_dev_emergency_ops`)
|
||||
_Archived 2026-06-19._
|
||||
|
||||
- `[2026-06-15]` **`.claude.json` (Claude Code config) truncates on ENOSPC → invalid JSON → CC resets to defaults on next start.** nh3-dev hit 100% disk (a peer's R19 dataset pull staged ~147G to local /home instead of the 7TB NAS); the in-flight `.claude.json` write was cut mid-key. Repair = strip the dangling token + close the open braces, validate `json.load`. CC keeps its OWN rolling backups at `~/.claude/backups/` + writes a `.claude.json.corrupted.*` snapshot when it self-detects. After ANY ENOSPC on a box running CC/agents: integrity-check `.claude.json` (json.load) AND the althing bus DB (`PRAGMA integrity_check`).
|
||||
_Archived 2026-06-19._
|
||||
|
||||
- `[2026-06-15]` **`pkill -f althing-light-monitor` SELF-MATCHES the killing shell** (the pattern is in the command's own argv) → kills itself mid-run (exit 144/truncated output). Stop the light-monitor via `althing-cli stop-monitor` or a captured PID — never `pkill -f <string-present-in-this-command>`. The singleton lock can also RACE to 2 live monitors during re-arm churn; keep exactly one tracked (run_in_background) monitor, and a raw `&` monitor is untracked (no harness fire-notification — don't use it).
|
||||
_Archived 2026-06-19._
|
||||
|
||||
- `[2026-06-14]` **Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP.** `gitea.phasefinal.com` = public `38.120.12.44` (ana-srv1); gitea is a container on ana-docker, git-SSH `10.250.50.70:222` + HTTP `:3000`. A fleet host egressing to public `:22` gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (`git fetch` times out under `set -euo pipefail`, aborts before reset). Bit irv-ml1's arbo deploy. `:22` on `10.250.50.70` is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented `docs/orientation.md` (`6e58e57`).
|
||||
_Archived 2026-06-19._
|
||||
|
||||
+2
-36
@@ -297,31 +297,9 @@ _As of 2026-06-19:_
|
||||
BEFORE the deploy image-swap restart, so new agents + new config land together (the
|
||||
v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread `01KV6X0M…`)
|
||||
|
||||
- `[2026-06-15]` **qwen36 thinking-split at the gateway** (operator) — base
|
||||
`qwen3.6-35b-a3b` = thinking-OFF default (the vision judge needs it), `…-a3b-thinking`
|
||||
= opt-in reasoning. One hybrid checkpoint + `--reasoning-parser qwen3`; mirrors the
|
||||
glm-5.1 split. The judge's enable_thinking=false is what makes the NVFP4 swap
|
||||
transparent. (`6de0844`)
|
||||
|
||||
- `[2026-06-15]` **arbo engine rebuilt v0.11.6 → v0.11.22 from main; catalog re-applied (v0.11.23) — curated `/workflows` footer live.** src/ schema changes (Model/Lora.summary, Workflow.blurb/keywords + the /workflows endpoint) need a MANUAL image rebuild (the webhook warns-only on `src/`/`Dockerfile`). Built from `/worktank/arbo/repo`, bumped `ARBO_IMAGE` in the host .env, recreated + verified healthy (auto-rollback armed, unneeded). comfy-dev's v0.11.20 catalog push had first crash-looped prod (old baked schema `extra=forbid` vs new git-mounted catalog fields → ValidationError); they reverted the catalog, so rebuild-from-main loaded clean, then they re-applied (v0.11.23). comfyui is a SEPARATE compose project — the rebuild didn't touch it.
|
||||
|
||||
- `[2026-06-15]` **althing-core refreshed to v0.14.1 box-wide on nh3-dev** (forseti + heid both routed it; central-sequenced per DevOps-delegation since `uv tool install --force` swaps the shared binary all handles use). Fixes the monitor non-truncating-lock bug so `stop-monitor`/`--stop` reliably recovers orphans. The editable install (`uv tool install --force --editable .`) covers every nh3-dev handle; `--version` trued up 0.14.0→0.14.1; verified `--stop` works against a real orphan.
|
||||
|
||||
- `[2026-06-15]` **ratatoskr Tier-3 affect live smoke GREEN** (consumer-confirmed). Brokered/minted consumer_id="ratatoskr" Heimdall HS256 key on personal Worldtree (the key IS BOTH the create-session Bearer AND ratatoskr's JWT verifier secret — plain utf-8 string, NOT base64/raw); injected ENV-ONLY to `~/.config/ratatoskr/provider.env` (mode 600) on nh3-dev; added `10.100.10.50:8390` to personal's `BIFROST_CLIENT_ALLOWED_HOSTS`; drove define→session (handshake 200)→turn→affect.emit (200, snapshot persisted). Worldtree signs Bifrost with the caller's raw Bearer verbatim. Drive commands + claims contract in althing thread `01KV4ZYSE3…`.
|
||||
|
||||
- `[2026-06-15]` **Dense Qwen3-VL-32B-NVFP4 A/B'd as arbo's anatomy hero-judge vs the qwen3.6-35b-a3b MoE → dense LOST, pilot torn down, GPU 0 reclaimed.** Dense NVFP4 loads clean on vLLM where the MoE NVFP4 doesn't (no expert-scale keys); ~44 tok/s on Blackwell GPU 0; vision tower stays high-precision. But comfy-dev's ground-truth A/B: MoE wins (faster + dual-use as general LLM + slight fine-detail edge), and the subtle-proportion tier is below ALL VLs. **Don't re-propose a dense-VL judge swap.** (auto-memory `project-qwen3vl-dense-pilot`)
|
||||
|
||||
- `[2026-06-15]` **MastMed prototype containerized + exposed publicly via cloudflared.** Self-contained static SPA (bun→caddy multi-stage build), basic-auth gated (operator's gating call), ana-docker:8089, cloudflared quick tunnel. See in-flight for the ephemeral-URL caveat + named-tunnel upgrade path.
|
||||
|
||||
- `[2026-06-15]` **infra-ops bootstrapped onto corviduo-dev** (operator-directed) — infra-ops now has SSH + NOPASSWD sudo there, OVERRIDING the fleet bootstrap script's deliberate partner-box exclusion (explicit host arg). corviduo config edits no longer need `vh` + a sudo password. (auto-memory `reference_corviduo_dev_emergency_ops`)
|
||||
|
||||
- `[2026-06-15]` **R18 clip+caption placement = ana-ml2** (operator: "ana-docker… ana-ml2 unless irv-ml1 has room"; irv-ml1 lacked free VRAM for the 30B-A3B captioner). Models staged + stub deployed; see in-flight + the hard real-voice gate.
|
||||
|
||||
- `[2026-06-14]` **LiteLLM infra-ops admin key provisioned** (operator) — resolves the LiteLLM half of the credential-migration directive; use it for ALL gateway ops (NOT `sk-corvid`). Value at `~/.config/litellm/infra-ops-key` (mode 600); gateway reachable directly from nh3-dev at 10.250.50.70:4000. (auto-memory `reference_litellm_infra_ops_key`)
|
||||
|
||||
- `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.** Agents currently reuse the operator's PERSONAL creds for infra ops — vh Gitea **admin** via `tea` (used this session to mint a `read:package` token for ratatoskr, id 13, **under vh**), `sk-corvid` litellm master for vkey admin. Stand up service accounts (a `claude-bot` Gitea user + scoped tokens, a distinct litellm admin key); re-mint consumer creds under them; flag personal-cred fallbacks until done. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
|
||||
|
||||
_79 older entries archived to archival-memory.md._
|
||||
_88 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
@@ -382,16 +360,4 @@ _79 older entries archived to archival-memory.md._
|
||||
("No available memory for the cache blocks"). Target one: `docker compose up -d <service>`.
|
||||
(auto-memory `feedback_compose_up_recreates_whole_stack`)
|
||||
|
||||
- `[2026-06-15]` **A comfyui restart-loop with `exit 0` + `OOMKilled=false` is NOT OOM — suspect a native SEGFAULT.** `docker inspect` shows only the LAST exit code; a GPU OOM is a catchable `RuntimeError: CUDA out of memory`, NEVER `Fatal Python error: Segmentation fault`. Diagnostic: `RestartCount>0` = same container restarted in-place → the docker json-log ACCUMULATES across all cycles, so grep the full `docker logs` for the crash signature + the lines right before each restart banner. comfyui's was at VAE decode (sampling completes, then segfault loading `AutoencodingEngine`) — gen-triggered but native, not VRAM. Suspects logged: torch/cu130 mismatch, an injected `aimdo` cuda-funchooks lib, the VAE.
|
||||
|
||||
- `[2026-06-15]` **NEVER blanket-kill `althing-light-monitor` by PID** — they're per-handle and MANY agents run their own on the shared nh3-dev box (heid / ratatoskr-dev / worldtree-dev / bifrost-dev each have one). A `kill $(pgrep -f light-monitor)` downs four peers' monitors (near-miss caught by checking `/proc/<pid>/environ` `ALTHING_HANDLE` + cwd first). Recover your OWN via `althing-cli stop-monitor` (v0.14.1+); a raw `&`-backgrounded monitor is untracked (no harness fire-notification) — always arm via `run_in_background`. (extends the pkill-self-match lesson below)
|
||||
|
||||
- `[2026-06-15]` **Manual `docker compose up` on corviduo Worldtree DOWNGRADES to a stale local `:latest` → crash-loop (config/image mismatch).** Compose `image:` is `…/worldtree:latest`, but local `:latest` lagged ~2wk (`21c90d5eeb2d`); CI/CD deploys NEWER SHA images, and a real pull 401s (daemon has no gitea-registry creds — that's CI/CD's). `--pull never` then used the stale latest → personal crash-looped on `ConfigurationError: echo.default_model missing` → **~2-3 min personal outage**. RECOVER: retag `:latest` to the image a HEALTHY SIBLING runs (`docker inspect worldtree-worldtree-api-1 --format '{{.Image}}'` → `docker tag …`), then `up -d --pull never`; verify `Application startup complete`. Don't manually recreate Worldtree on corviduo without pinning the correct image — image+config are CI/CD-coupled. (auto-memory `reference_corviduo_dev_emergency_ops`)
|
||||
|
||||
- `[2026-06-15]` **`.claude.json` (Claude Code config) truncates on ENOSPC → invalid JSON → CC resets to defaults on next start.** nh3-dev hit 100% disk (a peer's R19 dataset pull staged ~147G to local /home instead of the 7TB NAS); the in-flight `.claude.json` write was cut mid-key. Repair = strip the dangling token + close the open braces, validate `json.load`. CC keeps its OWN rolling backups at `~/.claude/backups/` + writes a `.claude.json.corrupted.*` snapshot when it self-detects. After ANY ENOSPC on a box running CC/agents: integrity-check `.claude.json` (json.load) AND the althing bus DB (`PRAGMA integrity_check`).
|
||||
|
||||
- `[2026-06-15]` **`pkill -f althing-light-monitor` SELF-MATCHES the killing shell** (the pattern is in the command's own argv) → kills itself mid-run (exit 144/truncated output). Stop the light-monitor via `althing-cli stop-monitor` or a captured PID — never `pkill -f <string-present-in-this-command>`. The singleton lock can also RACE to 2 live monitors during re-arm churn; keep exactly one tracked (run_in_background) monitor, and a raw `&` monitor is untracked (no harness fire-notification — don't use it).
|
||||
|
||||
- `[2026-06-14]` **Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP.** `gitea.phasefinal.com` = public `38.120.12.44` (ana-srv1); gitea is a container on ana-docker, git-SSH `10.250.50.70:222` + HTTP `:3000`. A fleet host egressing to public `:22` gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (`git fetch` times out under `set -euo pipefail`, aborts before reset). Bit irv-ml1's arbo deploy. `:22` on `10.250.50.70` is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented `docs/orientation.md` (`6e58e57`).
|
||||
|
||||
_70 older entries archived to archival-memory.md._
|
||||
_76 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user