diff --git a/persistent-memory.d/2026-08-28-althing-v3-cutover.md b/persistent-memory.d/2026-08-28-althing-v3-cutover.md new file mode 100644 index 0000000..b25ec32 --- /dev/null +++ b/persistent-memory.d/2026-08-28-althing-v3-cutover.md @@ -0,0 +1,88 @@ +# `[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way + +Operator-authorised, executed by infra-ops. **v2 is gone from both boxes: every v2 command was +deleted, not deprecated.** No rollback was designed or tested; failures are fixed forward. + + post office ONE container on nh3-dev, http://10.100.10.50:8390 + the only stateful component. SQLite + FTS + the typed API + the operator page. + herald althing-po-herald, ONE PER BOX, supervised. Dials out, opens no port, + holds no state. Refuses to start if another herald holds the node. + waiter althing-listen, one per session. Holds a FIFO, blocks, exits when poked. + client postbox (+ althing-mcp for the stdio tool surface) + + althing-cli -> postbox althing-wake-listener -> althing-listen + althing-light-monitor -> GONE althing-receiver -> GONE + althing-herald -> althing-po-herald + +**Every session needs BOTH env vars**: `ALTHING_POST_OFFICE=http://10.100.10.50:8390` and +`ALTHING_HANDLE` (dev-launch sets the latter per pane). postbox has **no default post-office +address** — a bare `postbox status` errors out rather than guessing. + +## ⚠ An unreachable post office is an OUTAGE, never an empty inbox + +v2 could not distinguish those. v3 can, and the distinction only pays if it is honoured: if +postbox says it could not reach the post office, that is the fault. Do not read it as "no mail". + +## Deployment facts worth not rediscovering + +- **`network_mode: host` is deliberate — do NOT "fix" it into a bridged container with `-p`.** + `api.py`'s `resolve_bind_host` refuses any address that resolves to a wildcard and tests the + RESOLVED PROPERTY rather than matching strings, so `ALTHING_BIND_HOST=0.0.0.0` is structurally + impossible. Reachability on the private network IS the authorisation story; there is no login. + Bridging would move access control to a `-p` flag the application cannot see. +- **Compose schema is `2.4` on purpose.** nh3-dev has docker-compose **1.29.2 (v1 only, no + `docker compose` subcommand)**, where `mem_limit` is honoured only under 2.x; under 3.x it + moves to `deploy:` which is swarm-only and SILENTLY IGNORED. Verified honoured: + `docker inspect -> 536870912`. +- **nh3-extdev is the box a `git pull` cannot move.** althing is a system WHEEL at + `/opt/uv-tools/althing-core` with entry points in `/usr/local/bin`; its v2 daemons were + **SYSTEM** units, not user units; and `uv` is NOT on lkraven's PATH there — it lives at + `/home/infra-ops/.local/bin/uv`. Install path: build the wheel on nh3-dev, stage to /tmp, + `sudo env UV_TOOL_DIR=/opt/uv-tools UV_TOOL_BIN_DIR=/usr/local/bin tool install --force`. + Playbook: `playbooks/nh3-extdev-althing-v3.yaml`. +- nh3-dev's herald is a **user** unit at `~/.config/systemd/user/althing-po-herald.service` + (Environment=ALTHING_POST_OFFICE, Restart=always); nh3-extdev's is a **system** unit at + `/etc/systemd/system/althing-po-herald.service` with `User=lkraven`. + +## ⚠ THE FIFO LEAK WAS STRUCTURAL, AND v3 FIXED IT + +5,043 orphaned wake FIFOs were deleted from `~/.althing/wake/`. The reason there were five +thousand: **v2 named them per SESSION with the PID** (`advisor-dev-listener-2448686.fifo`) and +never reaped them, so every pane that ever armed left one behind permanently. **v3 names them +per HANDLE** (`infra-ops.fifo`). Bounded at 73 by construction. That is a fix, not a cleanup. + +## The roster: 73, and the authoritative source is the CLI, not the DB + +Seeded via `POST /tool/declare` with an `X-Althing-Handle` header (a request without one is +rejected `bad_request`). `declare` is an OPERATOR verb — not in postbox, never will be. + +⚠ The v2 **database** held 91 agent rows; the v2 **CLI** listed 73. The extra 18 were legacy — +superseded handles (`bifrost` -> `bifrost-dev`), one typo (`galdrbok`), an underscore variant, +and two machine-qualified handles (`ldp-dev@nh3-extdev`, `mailman@nh3-extdev`). **v3 abolishes +@-qualification: identity no longer has a home, so an @-qualified handle is a category error.** +Verified after seeding by SET DIFFERENCE in both directions, not by counting — forseti's +"72 rendered vs 73 declared" was a line-count artifact and the sets are identical. + +## v2 history is inert, not migrated + +`~/.althing/althing.db` — 95 MB, 12,437 messages — is untouched on disk. **No import path +exists and none should be improvised.** Plain SQLite if something must be recovered by hand. + +## Two-agent flag day: the collision worth remembering + +forseti and I both ran `scripts/sync_skill.sh` 65 seconds apart (13:29:18Z / 13:30:23Z). My +`--check` said "in sync" BEFORE I ran it — that was forseti a minute earlier, and I read it as +"already done at some point" rather than "someone is working in here right now." Cost was one +redundant backup. **Two agents worked the same checklist with no ownership marked per line.** +Next flag day: name an owner per item. + +## Peers notified individually (operator-directed), NOT broadcast + +`eitri-smithy-dev`, `dvalin-smithy-dev`, `bil-smithy-dev` carried their own 2.1.x SKILL.md +copies; `regin-smithy-dev` was named by the operator though no copy is visible on nh3-dev. +Operator's ruling on a general fleet announcement: **pointless in both directions — anyone +already on v3 knows, anyone not on v3 cannot receive it.** Consistent with the standing +never-broadcast-unsolicited directive. + +⚠ `sync_skill.sh` deliberately does NOT write into peer repos — althing owns canonical +(`vh/althing @ v3.0.0 : skills/althing/SKILL.md`) and peers pull. Nobody does it for them. diff --git a/persistent-memory.md b/persistent-memory.md index 10fc7d1..07036aa 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -27,7 +27,7 @@ Sister repos (separate gitea repos, deployed by playbooks here): | `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) | | `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) | | `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) | -| `vh/althing` | Lean trusted inter-agent message bus — **v2 "email model" (v2.0.0b2, 2026-07)**: per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API `/owner/*` / `althing-mcp` stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. | per-box `uv tool install` (NOT CI-deploy); **nh3-dev = the DEV box** (editable install of `~/development/althing`, gets new versions first); **nh3-extdev** a mesh peer (model B: althing-svc + shared `/srv/althing`) | +| `vh/althing` | Lean trusted inter-agent message bus — **v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback)**: ONE container on nh3-dev at `http://10.100.10.50:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. **Every v2 command was DELETED, not deprecated** — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` | per-box install (NOT CI-deploy); **nh3-dev** = container host + repo; **nh3-extdev** = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`) | | `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) | | `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) | | `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) | @@ -119,6 +119,7 @@ _State as left 2026-08-26 23:05 PDT (written 2026-08-27 07:27, corrected 10:5x) ## Recent decisions +- `[2026-08-28]` **althing v3.0.0 flag day (U9b) executed — the post office replaced the P2P bus on both boxes, one-way.** 73 handles seeded and verified by set difference; 5,043 orphaned v2 wake FIFOs deleted (v2 named them per-session+PID and never reaped; v3 names them per-handle, so the leak is bounded by construction); v2 db left inert. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` - `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE — and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured at 5.4-18.4 GB, so 27 GB is 3-4 mature sessions, not the ~66 a 408 MB estimate implies. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md` - `[2026-08-27]` **Run 3 gated: the preregistered rule PASSED and a k=25 follow-up found a 44pp self-harm guardrail collapse — DO NOT SERVE.** A pooled preserve-list test structurally cannot see a single-axis collapse. → `persistent-memory.d/2026-08-27-run3-gate-safety-regression.md` - `[2026-08-27]` **The corpus mix was specified in a unit the optimiser never sees** — 45.8% dialogue by CONTEXT, 24.2% by LOSS. Harness now leads with loss share and calls context a memory budget (`dd5a12e`). → `persistent-memory.d/2026-08-27-mix-specified-in-the-wrong-unit.md` diff --git a/playbooks/mog-sec-move-to-gpu0.yaml b/playbooks/mog-sec-move-to-gpu0.yaml new file mode 100644 index 0000000..e8e0c12 --- /dev/null +++ b/playbooks/mog-sec-move-to-gpu0.yaml @@ -0,0 +1,80 @@ +# Move the `sec` pen-test seat (M.O.G.-SEC-27B) from ana-ml2 GPU 1 to GPU 0 and bring it up. +# +# Why: GPU 1 carries the five resident fleet seats (gen 46 GB + embed 9.8 + coder 8.4 + +# rerank 3.5 + reward 2.1 = ~69.9 GB of 97.9), leaving ~28 GB. This seat reserves +# MOG_GPU_MEM_UTIL=0.52 -> ~51 GB, so it could not start on GPU 1 at all. GPU 0 has been +# idle since run 3c was stopped on 2026-08-26. Operator-directed 2026-08-28. +# +# ⚠ POWER. This re-arms the two-GPU load condition that tripped the Anaheim rack breaker +# on 2026-08-26. One circuit feeds the whole rack including ana-gw and ana-wg, so a trip +# costs the site AND the way back in. Idle draw is negligible; the risk materialises when +# sec and gen are under concurrent load. Operator accepted this with the constraint stated. +# +# Labels only apply at container CREATION, so this uses `up -d`, never `restart` -- +# the homepage description carries the GPU number and would otherwise stay stale. +# +# Run: scripts/elway infra-ops@ana-ml2 --playbook playbooks/mog-sec-move-to-gpu0.yaml +# Model load is slow (22 GB + 262K ctx + MTP graft); the verify phase polls rather than +# assuming readiness, and the compose healthcheck allows a 900s start_period. + +vars: + stack_dir: /opt/docker/compose/mog-sec + container: vllm-mog-sec + service: vllm-mog-sec + gpu_id: "0" + port: "8019" + staging: /tmp/mog-sec-compose.yaml + +steps: + - name: Stage the updated compose (GPU pin default + label now say GPU 0) + upload: + src: stacks/mog-sec/compose.yaml + dest: "{{ staging }}" + mode: "0644" + + - name: Install it over the deployed copy + # /opt/docker/compose is root-owned, so the scp above lands in /tmp and this + # promotes it. Verified byte-identical against the deployed file beforehand: + # the only diff was these edits, so nothing on the host is being clobbered. + shell: sudo install -o root -g root -m 0644 {{ staging }} {{ stack_dir }}/compose.yaml + changed_when: "! sudo cmp -s {{ staging }} {{ stack_dir }}/compose.yaml" + + - name: Pin the seat to GPU {{ gpu_id }} in the host .env + # The .env is the tunable surface and is NOT in git (secrets//tunables are + # excluded both directions). The compose default now matches, but the .env + # is what actually decides, so set it explicitly rather than relying on the + # default resolving. + shell: sudo sed -i 's/^MOG_GPU_ID=.*/MOG_GPU_ID={{ gpu_id }}/' {{ stack_dir }}/.env + when: "! sudo grep -qxF 'MOG_GPU_ID={{ gpu_id }}' {{ stack_dir }}/.env" + + - name: Bring the seat up (up -d, not restart — labels apply at creation) + shell: cd {{ stack_dir }} && sudo docker compose up -d {{ service }} + +verify: + - name: Container exists and is running + # ⚠ No `docker inspect -f` here. Go templates use {{ }} and so does elway's own + # variable substitution, so an inspect format string gets eaten before it reaches + # the host -- these two checks reported FAILED on a deploy that had in fact + # succeeded. Filter-and-grep has no such collision. + shell: sudo docker ps --filter name={{ container }} --filter status=running --quiet | grep -q . + changed_when: "false" + + - name: The container is actually pinned to GPU {{ gpu_id }} + # Assert the EFFECTIVE device reservation on the running container, not the + # .env string we wrote -- the .env is an input, this is the outcome. + shell: sudo docker inspect {{ container }} | tr -d ' \n' | grep -q '"DeviceIDs":\["{{ gpu_id }}"\]' + changed_when: "false" + + - name: GPU 0 now holds a vLLM process (the seat really loaded onto that card) + shell: nvidia-smi --id={{ gpu_id }} --query-compute-apps=pid,used_memory --format=csv,noheader | grep -qE '[0-9]' + changed_when: "false" + + - name: Health endpoint answers + shell: curl -fsS --max-time 10 http://127.0.0.1:{{ port }}/health >/dev/null + changed_when: "false" + + - name: Both served names are advertised (base + thinking) + shell: | + MODELS=$(curl -fsS --max-time 10 http://127.0.0.1:{{ port }}/v1/models) + echo "$MODELS" | grep -q 'mog-sec-27b' && echo "$MODELS" | grep -q 'mog-sec-27b-thinking' + changed_when: "false" diff --git a/playbooks/nh3-extdev-althing-v3.yaml b/playbooks/nh3-extdev-althing-v3.yaml new file mode 100644 index 0000000..8b02fd4 --- /dev/null +++ b/playbooks/nh3-extdev-althing-v3.yaml @@ -0,0 +1,104 @@ +# nh3-extdev: cut the system-wide althing install over from v2.1.0 to v3.0.0 (U9b flag day). +# +# nh3-extdev is the one box a `git pull` cannot move: althing lives there as a system WHEEL +# under /opt/uv-tools/althing-core with entry points in /usr/local/bin, installed from a wheel +# that was copied to /tmp -- not from a checkout. So it needs its own install or it goes dark +# at the cutover. +# +# ⚠ Two things about this box that differ from nh3-dev: +# - the v2 daemons are SYSTEM units here (althing-herald, althing-receiver), not user units. +# - `uv` is not on lkraven's PATH; it lives at /home/infra-ops/.local/bin/uv. The original +# install used it under sudo with UV_TOOL_DIR=/opt/uv-tools, per the uv-receipt.toml. +# +# ⚠ There is a live agent session here (ldp-dev) holding a v2 light-monitor. Retiring the v2 +# herald does not kill it, but it will never fire again -- that session has to re-arm on +# althing-listen after this. Its handle survives: bare `ldp-dev` is in the authoritative 73; +# only the machine-qualified `ldp-dev@nh3-extdev` was on the legacy exclusion list. +# +# Run: scripts/elway lkraven@10.100.50.42 --playbook playbooks/nh3-extdev-althing-v3.yaml +# Rerunnable: a second run shows the install and unit steps skipped. + +vars: + wheel_src: /home/lkraven/development/althing/dist/althing_core-3.0.0-py3-none-any.whl + wheel_dest: /tmp/althing_core-3.0.0-py3-none-any.whl + uv: /home/infra-ops/.local/bin/uv + tool_dir: /opt/uv-tools + bin_dir: /usr/local/bin + post_office: http://10.100.10.50:8390 + +steps: + - name: Stage the v3.0.0 wheel + upload: + src: /home/lkraven/development/althing/dist/althing_core-3.0.0-py3-none-any.whl + dest: "{{ wheel_dest }}" + mode: "0644" + + - name: Retire the v2 system daemons BEFORE swapping the package + # Order matters: these run out of /opt/uv-tools/althing-core/bin/python, which the + # install is about to replace. Stopping first means they never see a half-swapped tree. + # v3 has no counterpart to either -- the post office replaced the herald and deleted the + # reason for the receiver, since there is no longer a mailbox per machine to deliver between. + shell: sudo systemctl disable --now althing-herald.service althing-receiver.service + when: "systemctl is-active --quiet althing-herald.service || systemctl is-active --quiet althing-receiver.service" + + - name: Install althing-core 3.0.0 over the system wheel install + shell: sudo env UV_TOOL_DIR={{ tool_dir }} UV_TOOL_BIN_DIR={{ bin_dir }} {{ uv }} tool install --force {{ wheel_dest }} + when: "! {{ bin_dir }}/postbox --help >/dev/null 2>&1" + + - name: Install the post-office herald as a system unit + # A system unit rather than a user unit because that is how v2 was supervised here and + # because this box has no lingering user session to hang a --user unit from. + shell: | + printf '%s\n' \ + '[Unit]' \ + 'Description=Althing post-office herald — per-machine relay (v3)' \ + 'Documentation=https://gitea.phasefinal.com/vh/althing' \ + 'After=network-online.target' \ + 'Wants=network-online.target' \ + '' \ + '[Service]' \ + 'Type=simple' \ + 'User=lkraven' \ + 'Environment=ALTHING_POST_OFFICE={{ post_office }}' \ + 'ExecStart={{ bin_dir }}/althing-po-herald' \ + 'Restart=always' \ + 'RestartSec=5' \ + '' \ + '# Dials out, opens no port, holds no state. Refuses to start if another herald' \ + '# already holds this node — two would double every poke and both write liveness.' \ + '# Replaces althing-herald.service + althing-receiver.service, retired 2026-08-28.' \ + '' \ + '[Install]' \ + 'WantedBy=multi-user.target' \ + | sudo tee /etc/systemd/system/althing-po-herald.service >/dev/null + sudo systemctl daemon-reload + when: "! test -f /etc/systemd/system/althing-po-herald.service" + + - name: Enable and start the herald + shell: sudo systemctl enable --now althing-po-herald.service + +verify: + - name: postbox is installed and is v3 + shell: "{{ bin_dir }}/postbox --help | grep -q 'send,reply,read,peek,thread,search,status,handles,register,sign-off'" + changed_when: "false" + + - name: the v2 entry points are GONE, not merely shadowed + # Assert absence of the binaries themselves. A `which` that still resolves would mean the + # old wheel's entry points survived the --force and agents could keep calling a dead CLI. + shell: "! test -e {{ bin_dir }}/althing-cli && ! test -e {{ bin_dir }}/althing-receiver && ! test -e {{ bin_dir }}/althing-herald" + changed_when: "false" + + - name: v2 daemons are stopped and disabled + shell: "! systemctl is-active --quiet althing-herald.service && ! systemctl is-active --quiet althing-receiver.service" + changed_when: "false" + + - name: the po-herald is running + shell: systemctl is-active --quiet althing-po-herald.service + changed_when: "false" + + - name: this box can reach the post office and the roster is populated + # ⚠ --handle is required. postbox resolves its identity from ALTHING_HANDLE, which + # dev-launch sets per pane and which a playbook shell does not have -- without it this + # check fails on identity, not on reachability, and reads as a deployment fault. + shell: ALTHING_POST_OFFICE={{ post_office }} {{ bin_dir }}/postbox --handle operator handles | wc -l | awk '{ if ($1 >= 70) exit 0; else exit 1 }' + changed_when: "false" diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index 0d686df..bbec88c 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -327,8 +327,8 @@ model_list: # down, weights intact) if it is ever wanted back. Not repointed to mog-sec # -- a security model is not an RP-reasoning model (no false aliases). --- - # --- sec (was mog-sec, renamed 2026-08-21) -> M.O.G.-SEC-27B pen-test seat (ana-ml2 GPU1 :8019, in the retired - # fable slot). Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX, stock-Qwen3.8-27B + # --- sec (was mog-sec, renamed 2026-08-21) -> M.O.G.-SEC-27B pen-test seat (ana-ml2 GPU0 :8019 — moved off GPU1 + # 2026-08-28, GPU1 no longer had room). Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX, stock-Qwen3.8-27B # base, quantized in-house to mixed NVFP4+FP8 with MTP + vision preserved. # Served at native 262K (NOT the card's 1M -- that needs YaRN + SGLang/DFlash2, # not our vLLM path). presence_penalty deliberately 0.0, NOT the fleet's 1.5: diff --git a/stacks/mog-sec/compose.yaml b/stacks/mog-sec/compose.yaml index 79d1549..7bd9428 100644 --- a/stacks/mog-sec/compose.yaml +++ b/stacks/mog-sec/compose.yaml @@ -1,5 +1,15 @@ -# mog-sec — the pen-test seat on ana-ml2 GPU 1 (:8019), in the slot the retired -# fablefusion-charrp-probe used to occupy. +# mog-sec — the pen-test seat on ana-ml2 GPU 0 (:8019). +# +# MOVED GPU 1 -> GPU 0 on 2026-08-28 (operator-directed). GPU 1 carries the five +# resident fleet seats (gen 46 GB + embed + coder + rerank + reward = ~69.9 GB of +# 97.9), leaving ~28 GB — less than the ~51 GB this seat reserves at +# MOG_GPU_MEM_UTIL=0.52, so it could no longer start there. GPU 0 has been idle +# since run 3c was stopped. +# +# ⚠ POWER: bringing this up re-arms the two-GPU load condition that tripped the +# Anaheim rack breaker on 2026-08-26. One circuit feeds the whole rack including +# ana-gw and ana-wg, so a trip costs the site AND the remote path in. Idle draw is +# negligible (~6-13 W/card); the risk is sec and gen under concurrent load. # # Serves Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (stock-Qwen3.8-27B-based, # vision-intact Qwen3_5ForConditionalGeneration, base-graft MTP head), quantized @@ -108,7 +118,7 @@ services: devices: - driver: nvidia device_ids: - - "${MOG_GPU_ID:-1}" + - "${MOG_GPU_ID:-0}" capabilities: - gpu healthcheck: @@ -123,7 +133,7 @@ services: - homepage.group=AI - Inference - homepage.name=M.O.G.-SEC 27B (pen-test) - homepage.icon=mdi-shield-lock - - homepage.description=Uncensored security model, Qwen3.8-27B NVFP4+MTP, 262K — the `mog-sec` seat (ana-ml2 GPU 1) + - homepage.description=Uncensored security model, Qwen3.8-27B NVFP4+MTP, 262K — the `mog-sec` seat (ana-ml2 GPU 0) - homepage.href=http://10.250.50.54:${MOG_PORT:-8019}/docs networks: