feat: althing v3.0.0 cutover (U9b) and the sec seat onto GPU0
Two operator-authorised changes on the same afternoon.
## althing v3 (U9b flag day, one-way, no rollback)
The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.
73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.
Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.
nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.
The v2 database is left inert on disk. There is no import path and none
was improvised.
## sec onto GPU0
GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.
Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
This commit is contained in:
@@ -0,0 +1,88 @@
|
||||
# `[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way
|
||||
|
||||
Operator-authorised, executed by infra-ops. **v2 is gone from both boxes: every v2 command was
|
||||
deleted, not deprecated.** No rollback was designed or tested; failures are fixed forward.
|
||||
|
||||
post office ONE container on nh3-dev, http://10.100.10.50:8390
|
||||
the only stateful component. SQLite + FTS + the typed API + the operator page.
|
||||
herald althing-po-herald, ONE PER BOX, supervised. Dials out, opens no port,
|
||||
holds no state. Refuses to start if another herald holds the node.
|
||||
waiter althing-listen, one per session. Holds a FIFO, blocks, exits when poked.
|
||||
client postbox (+ althing-mcp for the stdio tool surface)
|
||||
|
||||
althing-cli -> postbox althing-wake-listener -> althing-listen
|
||||
althing-light-monitor -> GONE althing-receiver -> GONE
|
||||
althing-herald -> althing-po-herald
|
||||
|
||||
**Every session needs BOTH env vars**: `ALTHING_POST_OFFICE=http://10.100.10.50:8390` and
|
||||
`ALTHING_HANDLE` (dev-launch sets the latter per pane). postbox has **no default post-office
|
||||
address** — a bare `postbox status` errors out rather than guessing.
|
||||
|
||||
## ⚠ An unreachable post office is an OUTAGE, never an empty inbox
|
||||
|
||||
v2 could not distinguish those. v3 can, and the distinction only pays if it is honoured: if
|
||||
postbox says it could not reach the post office, that is the fault. Do not read it as "no mail".
|
||||
|
||||
## Deployment facts worth not rediscovering
|
||||
|
||||
- **`network_mode: host` is deliberate — do NOT "fix" it into a bridged container with `-p`.**
|
||||
`api.py`'s `resolve_bind_host` refuses any address that resolves to a wildcard and tests the
|
||||
RESOLVED PROPERTY rather than matching strings, so `ALTHING_BIND_HOST=0.0.0.0` is structurally
|
||||
impossible. Reachability on the private network IS the authorisation story; there is no login.
|
||||
Bridging would move access control to a `-p` flag the application cannot see.
|
||||
- **Compose schema is `2.4` on purpose.** nh3-dev has docker-compose **1.29.2 (v1 only, no
|
||||
`docker compose` subcommand)**, where `mem_limit` is honoured only under 2.x; under 3.x it
|
||||
moves to `deploy:` which is swarm-only and SILENTLY IGNORED. Verified honoured:
|
||||
`docker inspect -> 536870912`.
|
||||
- **nh3-extdev is the box a `git pull` cannot move.** althing is a system WHEEL at
|
||||
`/opt/uv-tools/althing-core` with entry points in `/usr/local/bin`; its v2 daemons were
|
||||
**SYSTEM** units, not user units; and `uv` is NOT on lkraven's PATH there — it lives at
|
||||
`/home/infra-ops/.local/bin/uv`. Install path: build the wheel on nh3-dev, stage to /tmp,
|
||||
`sudo env UV_TOOL_DIR=/opt/uv-tools UV_TOOL_BIN_DIR=/usr/local/bin <uv> tool install --force`.
|
||||
Playbook: `playbooks/nh3-extdev-althing-v3.yaml`.
|
||||
- nh3-dev's herald is a **user** unit at `~/.config/systemd/user/althing-po-herald.service`
|
||||
(Environment=ALTHING_POST_OFFICE, Restart=always); nh3-extdev's is a **system** unit at
|
||||
`/etc/systemd/system/althing-po-herald.service` with `User=lkraven`.
|
||||
|
||||
## ⚠ THE FIFO LEAK WAS STRUCTURAL, AND v3 FIXED IT
|
||||
|
||||
5,043 orphaned wake FIFOs were deleted from `~/.althing/wake/`. The reason there were five
|
||||
thousand: **v2 named them per SESSION with the PID** (`advisor-dev-listener-2448686.fifo`) and
|
||||
never reaped them, so every pane that ever armed left one behind permanently. **v3 names them
|
||||
per HANDLE** (`infra-ops.fifo`). Bounded at 73 by construction. That is a fix, not a cleanup.
|
||||
|
||||
## The roster: 73, and the authoritative source is the CLI, not the DB
|
||||
|
||||
Seeded via `POST /tool/declare` with an `X-Althing-Handle` header (a request without one is
|
||||
rejected `bad_request`). `declare` is an OPERATOR verb — not in postbox, never will be.
|
||||
|
||||
⚠ The v2 **database** held 91 agent rows; the v2 **CLI** listed 73. The extra 18 were legacy —
|
||||
superseded handles (`bifrost` -> `bifrost-dev`), one typo (`galdrbok`), an underscore variant,
|
||||
and two machine-qualified handles (`ldp-dev@nh3-extdev`, `mailman@nh3-extdev`). **v3 abolishes
|
||||
@-qualification: identity no longer has a home, so an @-qualified handle is a category error.**
|
||||
Verified after seeding by SET DIFFERENCE in both directions, not by counting — forseti's
|
||||
"72 rendered vs 73 declared" was a line-count artifact and the sets are identical.
|
||||
|
||||
## v2 history is inert, not migrated
|
||||
|
||||
`~/.althing/althing.db` — 95 MB, 12,437 messages — is untouched on disk. **No import path
|
||||
exists and none should be improvised.** Plain SQLite if something must be recovered by hand.
|
||||
|
||||
## Two-agent flag day: the collision worth remembering
|
||||
|
||||
forseti and I both ran `scripts/sync_skill.sh` 65 seconds apart (13:29:18Z / 13:30:23Z). My
|
||||
`--check` said "in sync" BEFORE I ran it — that was forseti a minute earlier, and I read it as
|
||||
"already done at some point" rather than "someone is working in here right now." Cost was one
|
||||
redundant backup. **Two agents worked the same checklist with no ownership marked per line.**
|
||||
Next flag day: name an owner per item.
|
||||
|
||||
## Peers notified individually (operator-directed), NOT broadcast
|
||||
|
||||
`eitri-smithy-dev`, `dvalin-smithy-dev`, `bil-smithy-dev` carried their own 2.1.x SKILL.md
|
||||
copies; `regin-smithy-dev` was named by the operator though no copy is visible on nh3-dev.
|
||||
Operator's ruling on a general fleet announcement: **pointless in both directions — anyone
|
||||
already on v3 knows, anyone not on v3 cannot receive it.** Consistent with the standing
|
||||
never-broadcast-unsolicited directive.
|
||||
|
||||
⚠ `sync_skill.sh` deliberately does NOT write into peer repos — althing owns canonical
|
||||
(`vh/althing @ v3.0.0 : skills/althing/SKILL.md`) and peers pull. Nobody does it for them.
|
||||
@@ -27,7 +27,7 @@ Sister repos (separate gitea repos, deployed by playbooks here):
|
||||
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
|
||||
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
|
||||
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
|
||||
| `vh/althing` | Lean trusted inter-agent message bus — **v2 "email model" (v2.0.0b2, 2026-07)**: per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API `/owner/*` / `althing-mcp` stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. | per-box `uv tool install` (NOT CI-deploy); **nh3-dev = the DEV box** (editable install of `~/development/althing`, gets new versions first); **nh3-extdev** a mesh peer (model B: althing-svc + shared `/srv/althing`) |
|
||||
| `vh/althing` | Lean trusted inter-agent message bus — **v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback)**: ONE container on nh3-dev at `http://10.100.10.50:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. **Every v2 command was DELETED, not deprecated** — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` | per-box install (NOT CI-deploy); **nh3-dev** = container host + repo; **nh3-extdev** = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`) |
|
||||
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
|
||||
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
|
||||
| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) |
|
||||
@@ -119,6 +119,7 @@ _State as left 2026-08-26 23:05 PDT (written 2026-08-27 07:27, corrected 10:5x)
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-08-28]` **althing v3.0.0 flag day (U9b) executed — the post office replaced the P2P bus on both boxes, one-way.** 73 handles seeded and verified by set difference; 5,043 orphaned v2 wake FIFOs deleted (v2 named them per-session+PID and never reaped; v3 names them per-handle, so the leak is bounded by construction); v2 db left inert. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`
|
||||
- `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE — and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured at 5.4-18.4 GB, so 27 GB is 3-4 mature sessions, not the ~66 a 408 MB estimate implies. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md`
|
||||
- `[2026-08-27]` **Run 3 gated: the preregistered rule PASSED and a k=25 follow-up found a 44pp self-harm guardrail collapse — DO NOT SERVE.** A pooled preserve-list test structurally cannot see a single-axis collapse. → `persistent-memory.d/2026-08-27-run3-gate-safety-regression.md`
|
||||
- `[2026-08-27]` **The corpus mix was specified in a unit the optimiser never sees** — 45.8% dialogue by CONTEXT, 24.2% by LOSS. Harness now leads with loss share and calls context a memory budget (`dd5a12e`). → `persistent-memory.d/2026-08-27-mix-specified-in-the-wrong-unit.md`
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# Move the `sec` pen-test seat (M.O.G.-SEC-27B) from ana-ml2 GPU 1 to GPU 0 and bring it up.
|
||||
#
|
||||
# Why: GPU 1 carries the five resident fleet seats (gen 46 GB + embed 9.8 + coder 8.4 +
|
||||
# rerank 3.5 + reward 2.1 = ~69.9 GB of 97.9), leaving ~28 GB. This seat reserves
|
||||
# MOG_GPU_MEM_UTIL=0.52 -> ~51 GB, so it could not start on GPU 1 at all. GPU 0 has been
|
||||
# idle since run 3c was stopped on 2026-08-26. Operator-directed 2026-08-28.
|
||||
#
|
||||
# ⚠ POWER. This re-arms the two-GPU load condition that tripped the Anaheim rack breaker
|
||||
# on 2026-08-26. One circuit feeds the whole rack including ana-gw and ana-wg, so a trip
|
||||
# costs the site AND the way back in. Idle draw is negligible; the risk materialises when
|
||||
# sec and gen are under concurrent load. Operator accepted this with the constraint stated.
|
||||
#
|
||||
# Labels only apply at container CREATION, so this uses `up -d`, never `restart` --
|
||||
# the homepage description carries the GPU number and would otherwise stay stale.
|
||||
#
|
||||
# Run: scripts/elway infra-ops@ana-ml2 --playbook playbooks/mog-sec-move-to-gpu0.yaml
|
||||
# Model load is slow (22 GB + 262K ctx + MTP graft); the verify phase polls rather than
|
||||
# assuming readiness, and the compose healthcheck allows a 900s start_period.
|
||||
|
||||
vars:
|
||||
stack_dir: /opt/docker/compose/mog-sec
|
||||
container: vllm-mog-sec
|
||||
service: vllm-mog-sec
|
||||
gpu_id: "0"
|
||||
port: "8019"
|
||||
staging: /tmp/mog-sec-compose.yaml
|
||||
|
||||
steps:
|
||||
- name: Stage the updated compose (GPU pin default + label now say GPU 0)
|
||||
upload:
|
||||
src: stacks/mog-sec/compose.yaml
|
||||
dest: "{{ staging }}"
|
||||
mode: "0644"
|
||||
|
||||
- name: Install it over the deployed copy
|
||||
# /opt/docker/compose is root-owned, so the scp above lands in /tmp and this
|
||||
# promotes it. Verified byte-identical against the deployed file beforehand:
|
||||
# the only diff was these edits, so nothing on the host is being clobbered.
|
||||
shell: sudo install -o root -g root -m 0644 {{ staging }} {{ stack_dir }}/compose.yaml
|
||||
changed_when: "! sudo cmp -s {{ staging }} {{ stack_dir }}/compose.yaml"
|
||||
|
||||
- name: Pin the seat to GPU {{ gpu_id }} in the host .env
|
||||
# The .env is the tunable surface and is NOT in git (secrets//tunables are
|
||||
# excluded both directions). The compose default now matches, but the .env
|
||||
# is what actually decides, so set it explicitly rather than relying on the
|
||||
# default resolving.
|
||||
shell: sudo sed -i 's/^MOG_GPU_ID=.*/MOG_GPU_ID={{ gpu_id }}/' {{ stack_dir }}/.env
|
||||
when: "! sudo grep -qxF 'MOG_GPU_ID={{ gpu_id }}' {{ stack_dir }}/.env"
|
||||
|
||||
- name: Bring the seat up (up -d, not restart — labels apply at creation)
|
||||
shell: cd {{ stack_dir }} && sudo docker compose up -d {{ service }}
|
||||
|
||||
verify:
|
||||
- name: Container exists and is running
|
||||
# ⚠ No `docker inspect -f` here. Go templates use {{ }} and so does elway's own
|
||||
# variable substitution, so an inspect format string gets eaten before it reaches
|
||||
# the host -- these two checks reported FAILED on a deploy that had in fact
|
||||
# succeeded. Filter-and-grep has no such collision.
|
||||
shell: sudo docker ps --filter name={{ container }} --filter status=running --quiet | grep -q .
|
||||
changed_when: "false"
|
||||
|
||||
- name: The container is actually pinned to GPU {{ gpu_id }}
|
||||
# Assert the EFFECTIVE device reservation on the running container, not the
|
||||
# .env string we wrote -- the .env is an input, this is the outcome.
|
||||
shell: sudo docker inspect {{ container }} | tr -d ' \n' | grep -q '"DeviceIDs":\["{{ gpu_id }}"\]'
|
||||
changed_when: "false"
|
||||
|
||||
- name: GPU 0 now holds a vLLM process (the seat really loaded onto that card)
|
||||
shell: nvidia-smi --id={{ gpu_id }} --query-compute-apps=pid,used_memory --format=csv,noheader | grep -qE '[0-9]'
|
||||
changed_when: "false"
|
||||
|
||||
- name: Health endpoint answers
|
||||
shell: curl -fsS --max-time 10 http://127.0.0.1:{{ port }}/health >/dev/null
|
||||
changed_when: "false"
|
||||
|
||||
- name: Both served names are advertised (base + thinking)
|
||||
shell: |
|
||||
MODELS=$(curl -fsS --max-time 10 http://127.0.0.1:{{ port }}/v1/models)
|
||||
echo "$MODELS" | grep -q 'mog-sec-27b' && echo "$MODELS" | grep -q 'mog-sec-27b-thinking'
|
||||
changed_when: "false"
|
||||
@@ -0,0 +1,104 @@
|
||||
# nh3-extdev: cut the system-wide althing install over from v2.1.0 to v3.0.0 (U9b flag day).
|
||||
#
|
||||
# nh3-extdev is the one box a `git pull` cannot move: althing lives there as a system WHEEL
|
||||
# under /opt/uv-tools/althing-core with entry points in /usr/local/bin, installed from a wheel
|
||||
# that was copied to /tmp -- not from a checkout. So it needs its own install or it goes dark
|
||||
# at the cutover.
|
||||
#
|
||||
# ⚠ Two things about this box that differ from nh3-dev:
|
||||
# - the v2 daemons are SYSTEM units here (althing-herald, althing-receiver), not user units.
|
||||
# - `uv` is not on lkraven's PATH; it lives at /home/infra-ops/.local/bin/uv. The original
|
||||
# install used it under sudo with UV_TOOL_DIR=/opt/uv-tools, per the uv-receipt.toml.
|
||||
#
|
||||
# ⚠ There is a live agent session here (ldp-dev) holding a v2 light-monitor. Retiring the v2
|
||||
# herald does not kill it, but it will never fire again -- that session has to re-arm on
|
||||
# althing-listen after this. Its handle survives: bare `ldp-dev` is in the authoritative 73;
|
||||
# only the machine-qualified `ldp-dev@nh3-extdev` was on the legacy exclusion list.
|
||||
#
|
||||
# Run: scripts/elway lkraven@10.100.50.42 --playbook playbooks/nh3-extdev-althing-v3.yaml
|
||||
# Rerunnable: a second run shows the install and unit steps skipped.
|
||||
|
||||
vars:
|
||||
wheel_src: /home/lkraven/development/althing/dist/althing_core-3.0.0-py3-none-any.whl
|
||||
wheel_dest: /tmp/althing_core-3.0.0-py3-none-any.whl
|
||||
uv: /home/infra-ops/.local/bin/uv
|
||||
tool_dir: /opt/uv-tools
|
||||
bin_dir: /usr/local/bin
|
||||
post_office: http://10.100.10.50:8390
|
||||
|
||||
steps:
|
||||
- name: Stage the v3.0.0 wheel
|
||||
upload:
|
||||
src: /home/lkraven/development/althing/dist/althing_core-3.0.0-py3-none-any.whl
|
||||
dest: "{{ wheel_dest }}"
|
||||
mode: "0644"
|
||||
|
||||
- name: Retire the v2 system daemons BEFORE swapping the package
|
||||
# Order matters: these run out of /opt/uv-tools/althing-core/bin/python, which the
|
||||
# install is about to replace. Stopping first means they never see a half-swapped tree.
|
||||
# v3 has no counterpart to either -- the post office replaced the herald and deleted the
|
||||
# reason for the receiver, since there is no longer a mailbox per machine to deliver between.
|
||||
shell: sudo systemctl disable --now althing-herald.service althing-receiver.service
|
||||
when: "systemctl is-active --quiet althing-herald.service || systemctl is-active --quiet althing-receiver.service"
|
||||
|
||||
- name: Install althing-core 3.0.0 over the system wheel install
|
||||
shell: sudo env UV_TOOL_DIR={{ tool_dir }} UV_TOOL_BIN_DIR={{ bin_dir }} {{ uv }} tool install --force {{ wheel_dest }}
|
||||
when: "! {{ bin_dir }}/postbox --help >/dev/null 2>&1"
|
||||
|
||||
- name: Install the post-office herald as a system unit
|
||||
# A system unit rather than a user unit because that is how v2 was supervised here and
|
||||
# because this box has no lingering user session to hang a --user unit from.
|
||||
shell: |
|
||||
printf '%s\n' \
|
||||
'[Unit]' \
|
||||
'Description=Althing post-office herald — per-machine relay (v3)' \
|
||||
'Documentation=https://gitea.phasefinal.com/vh/althing' \
|
||||
'After=network-online.target' \
|
||||
'Wants=network-online.target' \
|
||||
'' \
|
||||
'[Service]' \
|
||||
'Type=simple' \
|
||||
'User=lkraven' \
|
||||
'Environment=ALTHING_POST_OFFICE={{ post_office }}' \
|
||||
'ExecStart={{ bin_dir }}/althing-po-herald' \
|
||||
'Restart=always' \
|
||||
'RestartSec=5' \
|
||||
'' \
|
||||
'# Dials out, opens no port, holds no state. Refuses to start if another herald' \
|
||||
'# already holds this node — two would double every poke and both write liveness.' \
|
||||
'# Replaces althing-herald.service + althing-receiver.service, retired 2026-08-28.' \
|
||||
'' \
|
||||
'[Install]' \
|
||||
'WantedBy=multi-user.target' \
|
||||
| sudo tee /etc/systemd/system/althing-po-herald.service >/dev/null
|
||||
sudo systemctl daemon-reload
|
||||
when: "! test -f /etc/systemd/system/althing-po-herald.service"
|
||||
|
||||
- name: Enable and start the herald
|
||||
shell: sudo systemctl enable --now althing-po-herald.service
|
||||
|
||||
verify:
|
||||
- name: postbox is installed and is v3
|
||||
shell: "{{ bin_dir }}/postbox --help | grep -q 'send,reply,read,peek,thread,search,status,handles,register,sign-off'"
|
||||
changed_when: "false"
|
||||
|
||||
- name: the v2 entry points are GONE, not merely shadowed
|
||||
# Assert absence of the binaries themselves. A `which` that still resolves would mean the
|
||||
# old wheel's entry points survived the --force and agents could keep calling a dead CLI.
|
||||
shell: "! test -e {{ bin_dir }}/althing-cli && ! test -e {{ bin_dir }}/althing-receiver && ! test -e {{ bin_dir }}/althing-herald"
|
||||
changed_when: "false"
|
||||
|
||||
- name: v2 daemons are stopped and disabled
|
||||
shell: "! systemctl is-active --quiet althing-herald.service && ! systemctl is-active --quiet althing-receiver.service"
|
||||
changed_when: "false"
|
||||
|
||||
- name: the po-herald is running
|
||||
shell: systemctl is-active --quiet althing-po-herald.service
|
||||
changed_when: "false"
|
||||
|
||||
- name: this box can reach the post office and the roster is populated
|
||||
# ⚠ --handle is required. postbox resolves its identity from ALTHING_HANDLE, which
|
||||
# dev-launch sets per pane and which a playbook shell does not have -- without it this
|
||||
# check fails on identity, not on reachability, and reads as a deployment fault.
|
||||
shell: ALTHING_POST_OFFICE={{ post_office }} {{ bin_dir }}/postbox --handle operator handles | wc -l | awk '{ if ($1 >= 70) exit 0; else exit 1 }'
|
||||
changed_when: "false"
|
||||
@@ -327,8 +327,8 @@ model_list:
|
||||
# down, weights intact) if it is ever wanted back. Not repointed to mog-sec
|
||||
# -- a security model is not an RP-reasoning model (no false aliases). ---
|
||||
|
||||
# --- sec (was mog-sec, renamed 2026-08-21) -> M.O.G.-SEC-27B pen-test seat (ana-ml2 GPU1 :8019, in the retired
|
||||
# fable slot). Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX, stock-Qwen3.8-27B
|
||||
# --- sec (was mog-sec, renamed 2026-08-21) -> M.O.G.-SEC-27B pen-test seat (ana-ml2 GPU0 :8019 — moved off GPU1
|
||||
# 2026-08-28, GPU1 no longer had room). Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX, stock-Qwen3.8-27B
|
||||
# base, quantized in-house to mixed NVFP4+FP8 with MTP + vision preserved.
|
||||
# Served at native 262K (NOT the card's 1M -- that needs YaRN + SGLang/DFlash2,
|
||||
# not our vLLM path). presence_penalty deliberately 0.0, NOT the fleet's 1.5:
|
||||
|
||||
@@ -1,5 +1,15 @@
|
||||
# mog-sec — the pen-test seat on ana-ml2 GPU 1 (:8019), in the slot the retired
|
||||
# fablefusion-charrp-probe used to occupy.
|
||||
# mog-sec — the pen-test seat on ana-ml2 GPU 0 (:8019).
|
||||
#
|
||||
# MOVED GPU 1 -> GPU 0 on 2026-08-28 (operator-directed). GPU 1 carries the five
|
||||
# resident fleet seats (gen 46 GB + embed + coder + rerank + reward = ~69.9 GB of
|
||||
# 97.9), leaving ~28 GB — less than the ~51 GB this seat reserves at
|
||||
# MOG_GPU_MEM_UTIL=0.52, so it could no longer start there. GPU 0 has been idle
|
||||
# since run 3c was stopped.
|
||||
#
|
||||
# ⚠ POWER: bringing this up re-arms the two-GPU load condition that tripped the
|
||||
# Anaheim rack breaker on 2026-08-26. One circuit feeds the whole rack including
|
||||
# ana-gw and ana-wg, so a trip costs the site AND the remote path in. Idle draw is
|
||||
# negligible (~6-13 W/card); the risk is sec and gen under concurrent load.
|
||||
#
|
||||
# Serves Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (stock-Qwen3.8-27B-based,
|
||||
# vision-intact Qwen3_5ForConditionalGeneration, base-graft MTP head), quantized
|
||||
@@ -108,7 +118,7 @@ services:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids:
|
||||
- "${MOG_GPU_ID:-1}"
|
||||
- "${MOG_GPU_ID:-0}"
|
||||
capabilities:
|
||||
- gpu
|
||||
healthcheck:
|
||||
@@ -123,7 +133,7 @@ services:
|
||||
- homepage.group=AI - Inference
|
||||
- homepage.name=M.O.G.-SEC 27B (pen-test)
|
||||
- homepage.icon=mdi-shield-lock
|
||||
- homepage.description=Uncensored security model, Qwen3.8-27B NVFP4+MTP, 262K — the `mog-sec` seat (ana-ml2 GPU 1)
|
||||
- homepage.description=Uncensored security model, Qwen3.8-27B NVFP4+MTP, 262K — the `mog-sec` seat (ana-ml2 GPU 0)
|
||||
- homepage.href=http://10.250.50.54:${MOG_PORT:-8019}/docs
|
||||
|
||||
networks:
|
||||
|
||||
Reference in New Issue
Block a user