diff --git a/dns/internal.yaml b/dns/internal.yaml index c9dc319..d2bbbe6 100644 --- a/dns/internal.yaml +++ b/dns/internal.yaml @@ -122,6 +122,7 @@ hosts: # runs becomes a one-line edit here instead of a hunt through configs. aliases: + - {name: albok, site: nh3, target: nh3-docker, note: albok-service (fleet knowledgebase) :8392 — container :8390; 8390 on the host is the post office} - {name: searxng, site: nh3, target: nh3-docker, note: moved off ana-docker 2026-09-03 — colo egress (38.120.12.42) is CAPTCHA-gated by search engines; NH3 egresses residentially} - {name: gateway, site: ana, target: ana-docker, note: LiteLLM gateway :4000} - {name: booth, site: nh3, target: nh3-dev, note: The Booth :8090} diff --git a/persistent-memory.md b/persistent-memory.md index 1385514..fd02cb3 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -290,6 +290,7 @@ _As of 2026-10-01 ~0446 PT._ - `[2026-10-02]` **Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684).** Their b193 release commit c2d87263 swept 60 staged deletions into the tree, so the demo api crash-looped. I recreated `worldtree-api` only (`compose up -d --no-deps`) on the `.env`-pinned fa8bc51cc064 (compose.yaml identical at both shas): healthy in 35 s at 15:16Z. ⚠ Their deploy health gate does NOT restore the old container on failure (it only withholds `:latest`); flagged to them. Fix-forward 7ab6ae40 (tag v1.0.0b193 moved onto it) deployed via CI 15:22Z and verified healthy, same seven retired-key WARNINGs; the deploy-gate gap is Worldtree #423. - `[2026-10-02]` **nh3-pve's AMT is in TacticalRMM's MeshCentral (group `PFI-AMT`, device `nh3-pve-amt`).** The adds had failed silently because TRMM installs MeshCentral `WANonly` (meshuser.js:2682 drops AMT adds). Prime ruled hybrid; switched 0812, 14/14 agents back, AMT connected at once (16.1.25, TLS fine). **UPDATE 0859: it now PHONES HOME (CIRA)** — MeshCentral mpsPass, ana-gw VIP/policy 76 on 4433, AMT settings via `scripts/amt-cira-setup.py`, AMT moved static → DHCP (Intel: CIRA needs DHCP; reservation keeps .61). Tunnel is independent of nh3-pve. ⚠ While phoning home the AMT REFUSES LAN management (:16993 dark), so manage it via MeshCentral; revert body `servers/nh3-pve/amt-ethernet-static-revert.xml`. 2FA still not forced. Prime's MeshCentral login token is vaulted `pfi-tacticalrmm/meshcentral-login-token`. → `servers/pfi-tacticalrmm/README.md` - `[2026-10-02]` **Prime: dev-backup gets dailies + weeklies — DONE.** Retention is now 48 hourly + newest of 30 days + newest of 12 ISO weeks (`retention.py` beside the script; second path guard on the NAS side; unit fails if kept ≠ expected). Live run 0739: deleted 1, 0 errors, 48 kept = expected. History before 09-30 was already gone; dailies accumulate from today. +- `[2026-10-02]` **albok-service 0.1.0 LIVE on nh3-docker (albok-dev ask, operator-approved): `http://albok.nh3.internal:8392`** (8390 there is the post office). Store/private on local ext4 under /srv/albok, uid 1500, read groups 1510/1511 (container gets a mounted /etc/group + group_add so its getgrnam/chgrp work). Scoped LiteLLM key `albok-service` (qwen3-embedding only) + bootstrap admin token in the vault under `albok/`. ⚠ 0.1.0 /health says `degraded` by a canary-vocabulary bug ('alive' not accepted) — reported. → `stacks/albok-service/README.md` - `[2026-10-02]` **nh3-pve-2 status at end of day: AMT done, Proxmox on the WRONG NVMe; Prime reinstalls ON SITE 2026-10-03.** IDE-R from MeshCentral (ISO `proxmox-ve_9.2-1.iso` in lkraven's My Files, kept) installed it but then stalled across the internet; the 4K dummy plug blacks the AMT console once Linux takes the display (8-bit/grayscale encoding or a 1080p plug fixes it); the Realtek 10G (`…:a0:a9`, USW port 1) only links during firmware, so it likely has no driver. **On-site checklist:** USB-stick install → pick the right disk under Target Harddisk → Options; host NIC = i226 (shared with AMT) or an X710 SFP+, not the Realtek; then I wipe the old disk (`wipefs` + `zpool labelclear` if ZFS: two `rpool`s collide), reserve .62, onboard `infra-ops`, keep the AMT port admin-UP in Linux (MS-01 lesson), swap in a 1080p plug. - `[2026-10-02]` **Prime: MS-03s get Proxmox; dummy plugs in hand. One MS-03 is being deployed as `nh3-pve-2` at NH3 (Prime, ~0910).** Plan: its AMT on DHCP (needed for phone-home) with a UDM reservation, proposed 10.100.250.63 / host 10.100.250.62 (static in PVE + reservation); configure KVM/opt-in over LAN FIRST, then `scripts/amt-cira-setup.py` (LAN goes dark after). MS-03 = i226-LM vPro 2.5G + RTL8127 10G RJ45 + 2× X710 SFP+. **Cabled 2026-10-02 1435 on UDM port 5: AMT 21.0.6 answers (TLS-only, :16993/:664), MAC 38:05:25:3b:a0:a6, sharing the factory Windows' DHCP address (WIN-94HJ50P1LUE).** Port 5 is now native nh3-mgmt (copy of port 6); reservation nh3-pve-2-amt = 10.100.250.63; DNS added. It moved to .63 at 14:47 (replug/reboot). Then (MEBx password = nh3-pve's, vaulted `nh3-pve-2/amt-admin`): KVM on, listener on, OptIn 0 (ACM), `amt-cira-setup.py --apply` → **phoning home at once**; MeshCentral device `nh3-pve-2-amt` (creds + tls=1, needed a MeshCentral restart to log in) shows AMT 21.0.6, power on. LAN :16993 dark by design. Remote install: MeshCentral's embedded MeshCommander (device → Intel AMT tab) has IDE-R; AMT_RedirectionService 32771 = IDER+SOL enabled. **Roles (Prime ~0915): both MS-03 run Proxmox. The other one hosts `esh-dev` at ESH, which will INHERIT MOST OF nh3-dev's SESSIONS (a migration, not yet planned). nh3-pve-2's purpose is TBD ON PURPOSE (high-powered PVE host; possibly a dev environment for security software).** Do not assign it a role. When the esh-dev move is planned, inventory what is anchored to nh3-dev first: the althing herald, svos/hermes-gateway (Miranda's channel), the Booth, the fleet TLS caddy and the `*.nh3.phasefinal.com` rewrite to 10.100.10.50, dev-backup, ttyd/zellij seats, and the `nh3-dev/` vault namespace. On arrival: check the NIC chipset (I226-LM = keep the AMT port admin-UP), fit a plug on each, then the parked AMT follow-ups. - `[2026-10-02]` **nh3-dev root grown 250 → 378 GB (Prime resized scsi0; I grew the guest online, no reboot).** Root 372 GB, 58%, 150 GB free, after the 85% alert fired twice in 14 h (uv cache + agent venvs). Swap moved to a 4 GB `/swapfile` (the old `sda5` blocked growth), `RESUME=none`, all initrds rebuilt (`playbooks/nh3-dev-grow-root.yaml`). ⚠ **TODO: delete VM snapshot `pre-rootgrow-20261002` on nh3-pve after the next NATURAL reboot (Prime 2026-10-02: no test reboot).** Trigger: `uptime -s` on nh3-dev later than 2026-10-02 0733. Then check the boot was clean (`swapon --show` = /swapfile; `systemd-analyze` shows no ~30 s stall; `journalctl -b | grep -i resume` has no 'waiting for resume device'), and only then `qm delsnapshot 102 pre-rootgrow-20261002`. diff --git a/playbooks/albok-service-host.yaml b/playbooks/albok-service-host.yaml new file mode 100644 index 0000000..d245a26 --- /dev/null +++ b/playbooks/albok-service-host.yaml @@ -0,0 +1,41 @@ +# Host preparation for albok-service on nh3-docker (2026-10-02). Idempotent. +# scripts/elway infra-ops@10.100.50.40 --playbook playbooks/albok-service-host.yaml +# Creates the service identity and read groups with FIXED numeric ids (the contract requires them), +# and the two local roots owned by uid 1500. The config file itself (it carries the embedder key) +# is uploaded separately to /srv/albok/etc/albok.yaml: see stacks/albok-service/README.md. +steps: + - name: group albok (gid 1500) + sudo: true + shell: groupadd --gid 1500 albok + when: "! getent group albok >/dev/null" + - name: user albok (uid 1500, no login, no home) + sudo: true + shell: useradd --system --uid 1500 --gid 1500 --no-create-home --home-dir /nonexistent --shell /usr/sbin/nologin albok + when: "! getent passwd albok >/dev/null" + - name: read group albok-read (gid 1510) — building `fleet` + sudo: true + shell: groupadd --gid 1510 albok-read + when: "! getent group albok-read >/dev/null" + - name: read group albok-personal (gid 1511) — building `personal` + sudo: true + shell: groupadd --gid 1511 albok-personal + when: "! getent group albok-personal >/dev/null" + - name: store root, private root and config dir (local ext4, never NFS) + sudo: true + shell: | + set -e + install -d -o 1500 -g 1500 -m 0711 /srv/albok/store + install -d -o 1500 -g 1500 -m 0700 /srv/albok/private + install -d -o root -g 1500 -m 0750 /srv/albok/etc + changed_when: "false" + +verify: + - name: ids are the fixed numbers + shell: | + test "$(getent passwd albok | cut -d: -f3,4)" = "1500:1500" && test "$(getent group albok-read | cut -d: -f3)" = 1510 && test "$(getent group albok-personal | cut -d: -f3)" = 1511 + changed_when: "false" + - name: roots are local (not NFS) and owned by 1500 + sudo: true + shell: | + test "$(findmnt -T /srv/albok/store -no FSTYPE)" = ext4 && test "$(stat -c %u /srv/albok/store)" = 1500 && test "$(stat -c %u /srv/albok/private)" = 1500 + changed_when: "false" diff --git a/stacks/albok-service/README.md b/stacks/albok-service/README.md new file mode 100644 index 0000000..b7614be --- /dev/null +++ b/stacks/albok-service/README.md @@ -0,0 +1,71 @@ +# albok-service — fleet knowledgebase service (nh3-docker) + +The FastAPI process that owns Albok's buildings-and-wings store (`vh/albok`, +`packages/albok-service`, contract `docs/contracts/unit2_service.contract.md`). It is the +**only writer** of the store. Deployed 2026-10-02 at albok-dev's request (operator-approved). + +- **URL:** `http://albok.nh3.internal:8392` (= `http://10.100.50.40:8392`). Container port 8390. + ⚠ Host port **8392**, because **8390 on nh3-docker is the althing post office**. +- **Image:** `gitea.phasefinal.com/pfi/albok-service:0.1.0` @ `sha256:9ae2c94e…` (pinned in + `compose.yaml`), built from vh/albok `0b37431` (tag v0.1.1; albok core 0.1.1 inside). +- **Health:** `GET /health` (no auth). ⚠ 0.1.0 reports `"status": "degraded"` even when everything + is fine: `health.py` treats canary values other than ok/healthy/ready as dead, and the canary + reports `"alive"`. Reported to albok-dev. Read the fields, not `status`. The Docker healthcheck + only checks for HTTP 200. + +## Pieces and where they live + +| What | Where | +|---|---| +| store root (one git repo per wing) | `/srv/albok/store` — local ext4, `albok:albok` 0711 | +| private root (lease, journal, tokens.db, per-wing chroma) | `/srv/albok/private` — local ext4, `albok:albok` 0700, **single-attach** | +| config (holds the embedder key) | `/srv/albok/etc/albok.yaml` — `root:albok` 0640; rendered from `conf/albok.yaml.template` | +| container `/etc/group` | `/opt/docker/conf/albok-service/group` (= `conf/group`) | +| compose | `/opt/docker/compose/albok-service/compose.yaml` | + +**Identities (fixed numeric ids, created by `playbooks/albok-service-host.yaml`):** user and group +`albok` 1500:1500 (the image's user), read groups `albok-read` **1510** (building `fleet`) and +`albok-personal` **1511** (building `personal`). Reserve 1512+ for restricted wings (agent-feedback, +vault) and add each to the host, `conf/group` and `group_add`. + +**Why the group file and `group_add`:** the service resolves group NAMES with `grp.getgrnam()` +inside the container and `chgrp`s wing dirs as uid 1500. Without the names in the container's +`/etc/group` it reports "group does not resolve", and without membership the chgrp is refused. +Verified: wings came up `drwxr-s--- albok:albok-read` / `albok:albok-personal`, drift 0. + +**Secrets (vault):** `albok/litellm-key`, a LiteLLM key scoped to `qwen3-embedding` only (alias +`albok-service`; verified 200 on embeddings, 403 on any other model); `albok/bootstrap-admin-token`, +the first-start admin token (also at `/srv/albok/private/bootstrap-admin-token`, 0600). + +**Backups:** nh3-docker is VM 100 on nh3-pve, in the nightly 21:00 vzdump (snapshot mode → pbs-ana), +so `/srv/albok` is covered whole-VM. No file-level restic job for it. + +## Deploy / redeploy + +```bash +# 1. host prep (idempotent) +scripts/elway infra-ops@10.100.50.40 --playbook playbooks/albok-service-host.yaml +# 2. config: render the template with the vaulted key (never commit the rendered file) +umask 077; secret get albok/litellm-key | tr -d '\n' | python3 -c "import sys; k=sys.stdin.read(); \ + open('/tmp/albok.yaml','w').write(open('stacks/albok-service/conf/albok.yaml.template').read().replace('__ALBOK_LITELLM_KEY__', k))" +scripts/elway infra-ops@10.100.50.40 --upload /tmp/albok.yaml:/srv/albok/etc/albok.yaml:0640 --sudo --owner root:albok +# then delete /tmp/albok.yaml by its literal path +# 3. compose + conf, then start (root holds the registry login on nh3-docker) +scripts/deploy-stack.sh nh3-docker albok-service +ssh infra-ops@10.100.50.40 'cd /opt/docker/compose/albok-service && sudo docker compose up -d' +``` + +**Image rebuild** (from a clean archive, never a working tree): + +```bash +git -C ~/development/albok archive | tar -x -C +cd && docker build -f packages/albok-service/Dockerfile -t gitea.phasefinal.com/pfi/albok-service: . +secret get nh3-dev/.config/claude-bot/gitea-token-sdkops | docker login gitea.phasefinal.com -u claude-bot --password-stdin +docker push gitea.phasefinal.com/pfi/albok-service: # then pin the new digest in compose.yaml +``` + +## Viewers (read-only access to the store) + +Mount `/srv/albok/store` **`:ro`**, `group_add` the read group's **numeric** gid (1510 / 1511), set +`GIT_OPTIONAL_LOCKS=0`, and add `safe.directory` for the path as the viewer sees it. Nothing else +may mount the store read-write. diff --git a/stacks/albok-service/compose.yaml b/stacks/albok-service/compose.yaml new file mode 100644 index 0000000..2e01508 --- /dev/null +++ b/stacks/albok-service/compose.yaml @@ -0,0 +1,48 @@ +# albok-service — the fleet knowledgebase service (vh/albok, packages/albok-service). The only +# writer of the buildings-and-wings store. Requested by albok-dev 2026-10-02 (operator-approved). +# +# Deploy: scripts/elway infra-ops@10.100.50.40 --playbook playbooks/albok-service-host.yaml (host prep, config) +# scripts/deploy-stack.sh nh3-docker albok-service (this file + conf/) +# Image: built from a clean `git archive` of vh/albok, pushed to the pfi org (see README.md). +# +# ⚠ Host port 8392, NOT 8390: 8390 on nh3-docker is the althing post office. +# ⚠ Exactly ONE instance per private root: a second one exits 75 on the lease. Never scale this. +# ⚠ Nothing else may mount /srv/albok/store read-write. Viewers mount it :ro with group_add by gid +# (1510 albok-read, 1511 albok-personal) and need GIT_OPTIONAL_LOCKS=0 + a safe.directory entry. +services: + albok-service: + image: gitea.phasefinal.com/pfi/albok-service:0.1.0@sha256:9ae2c94e39917d690aef2361e47abceff863f25338fd0ea2848b651bcb145ab6 + container_name: albok-service + restart: unless-stopped + ports: + - "8392:8390" + # Membership the service needs to chgrp wing dirs to the read groups (it runs as uid 1500, not root). + group_add: + - "1510" + - "1511" + volumes: + - /srv/albok/store:/srv/albok/store + - /srv/albok/private:/srv/albok/private + - /srv/albok/etc/albok.yaml:/etc/albok/albok.yaml:ro + # the read groups must RESOLVE BY NAME inside the container (layout.py uses grp.getgrnam) + - /opt/docker/conf/albok-service/group:/etc/group:ro + mem_limit: 4g + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8390/health', timeout=5).status == 200 else 1)"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + networks: + - tnet + labels: + - homepage.group=Agents (no UI) + - homepage.name=Albok + - homepage.icon=mdi-book-search + - homepage.description=Fleet knowledgebase service (albok-service) on nh3-docker + - homepage.href=http://10.100.50.40:8392/health + +networks: + tnet: + name: traefik-net + external: true diff --git a/stacks/albok-service/conf/albok.yaml.template b/stacks/albok-service/conf/albok.yaml.template new file mode 100644 index 0000000..3398696 --- /dev/null +++ b/stacks/albok-service/conf/albok.yaml.template @@ -0,0 +1,45 @@ +# albok-service configuration (template). The REAL file is /srv/albok/etc/albok.yaml on nh3-docker, +# rendered from this one with the embedder key filled in (vault `albok/litellm-key`) by +# playbooks/albok-service-host.yaml. It is not under /opt/docker/conf because it holds that key and +# deploy-stack would sync over it. Every path is absolute and used verbatim (nothing is expanded). +# Shape: packages/albok-service/albok.example.yaml in vh/albok @ 0b37431 (albok-service 0.1.0). + +store_root: /srv/albok/store # host bind mount, local ext4, owned by uid 1500 +private_root: /srv/albok/private # host bind mount, local ext4, owned by uid 1500; single-attach (lease) + +git_identity: + name: Albok + email: albok@fleet.local + +embedding: # fleet LiteLLM gateway (Anaheim); TEI behind it + provider: litellm-ana + type: openai_compat + base_url: http://10.250.50.70:4000/v1 + model: qwen3-embedding + api_key: "__ALBOK_LITELLM_KEY__" # scoped key `albok-service`: qwen3-embedding only + +reranker: null + +ids_in_cap: 1000 +search_defaults: + top_k: 20 + inline_budget_chars: 8000 +listen: + host: 0.0.0.0 + port: 8390 # container port; published on the host as 8392 (8390 is the post office) + +# First instance, per albok-dev's ask (2026-10-02): fleet/{memory,stash}, personal/notes. +# Restricted wings (agent-feedback, vault) come later with their own groups (gids 1512+). +buildings: + - id: fleet + default_group: albok-read # host gid 1510 + wings: + - id: memory + probe_object_id: null + probe_query: null + - id: stash + near_duplicate_threshold: null + - id: personal + default_group: albok-personal # host gid 1511 + wings: + - id: notes diff --git a/stacks/albok-service/conf/group b/stacks/albok-service/conf/group new file mode 100644 index 0000000..4042da1 --- /dev/null +++ b/stacks/albok-service/conf/group @@ -0,0 +1,41 @@ +root:x:0: +daemon:x:1: +bin:x:2: +sys:x:3: +adm:x:4: +tty:x:5: +disk:x:6: +lp:x:7: +mail:x:8: +news:x:9: +uucp:x:10: +man:x:12: +proxy:x:13: +kmem:x:15: +dialout:x:20: +fax:x:21: +voice:x:22: +cdrom:x:24: +floppy:x:25: +tape:x:26: +sudo:x:27: +audio:x:29: +dip:x:30: +www-data:x:33: +backup:x:34: +operator:x:37: +list:x:38: +irc:x:39: +src:x:40: +shadow:x:42: +utmp:x:43: +video:x:44: +sasl:x:45: +plugdev:x:46: +staff:x:50: +games:x:60: +users:x:100: +nogroup:x:65534: +albok:x:1500: +albok-read:x:1510:albok +albok-personal:x:1511:albok