diff --git a/persistent-memory.md b/persistent-memory.md index 30d7329..fb4e75b 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -287,6 +287,7 @@ _As of 2026-10-01 ~0446 PT._ ## Recent decisions +- `[2026-10-03]` **albok-service 0.1.2 LIVE on nh3-docker (albok-dev ask 6955, 0251), fixing a leak that had broken 0.1.1.** 0.1.1 leaked a Chroma client per health probe: after 9 h it held 1018 of 1024 fds and `/health`/`/search` answered 500. 0.1.2 (`sha256:6f91a33f…`, built from albok-dev's LOCAL `544ecfd` before the operator pushed it: verify origin's tag after the push) holds 59 fds, 0 deleted, over 30 health calls. Added restricted wing `personal/agent-feedback` (group `albok-feedback` 1512, default-deny). Nemi credentials: service token re-minted with agent-feedback grants (`albok/token-nemi`; old token still live until albok-dev revokes it) and LiteLLM key `albok-nemi` (gen-small/summarizer/qwen3-embedding; `albok/litellm-key-nemi`). **Nemi timer NOT installed:** albok-dev will confirm the steady-state walk time first, and the first 1 h 48 m ingest runs by hand. → `stacks/albok-service/README.md` - `[2026-10-02]` **esh-pve-2 (MS-03 at ESH, future `esh-dev` host) onboarded: infra-ops (Prime), 1 TB → LVM-thin `vmstore`, AMT phoning home as `esh-pve-2-amt`.** The stale firmware boot entry for an old install on the 1 TB was deleted; that drive was wiped and pvesh-created as `vmstore` (913 GiB; `playbooks/esh-pve-2-disk-prep.yaml`); the reboot onto kernel 7.0.14-20 is verified. AMT 21 (ACM, same MEBx pw, vaulted `esh-pve-2/amt-admin`) shares nic1 (I226-LM), its MAC and its DHCP IP with the host: KVM on, OptIn 0, listener on, then `amt-cira-setup.py --apply`. ⚠ Its FIRST CIRA connect crashed MeshCentral once (~7 s, agents back; MeshCentral 1.2.0 race in mpsserver.js, read at source); credentials were then picked up by dropping only its tunnel (`ss -K`), with no restart. Address 10.0.10.70 is TEMPORARY (Prime: can wait) and must become a reservation on MAC `38:05:25:3b:9c:12`. → `servers/esh-pve-2/README.md`, `servers/pfi-tacticalrmm/README.md` - `[2026-10-02]` **Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684).** Their b193 release commit c2d87263 swept 60 staged deletions into the tree, so the demo api crash-looped. I recreated `worldtree-api` only (`compose up -d --no-deps`) on the `.env`-pinned fa8bc51cc064 (compose.yaml identical at both shas): healthy in 35 s at 15:16Z. ⚠ Their deploy health gate does NOT restore the old container on failure (it only withholds `:latest`); flagged to them. Fix-forward 7ab6ae40 (tag v1.0.0b193 moved onto it) deployed via CI 15:22Z and verified healthy, same seven retired-key WARNINGs; the deploy-gate gap is Worldtree #423. - `[2026-10-02]` **nh3-pve's AMT is in TacticalRMM's MeshCentral (group `PFI-AMT`, device `nh3-pve-amt`).** The adds had failed silently because TRMM installs MeshCentral `WANonly` (meshuser.js:2682 drops AMT adds). Prime ruled hybrid; switched 0812, 14/14 agents back, AMT connected at once (16.1.25, TLS fine). **UPDATE 0859: it now PHONES HOME (CIRA)** — MeshCentral mpsPass, ana-gw VIP/policy 76 on 4433, AMT settings via `scripts/amt-cira-setup.py`, AMT moved static → DHCP (Intel: CIRA needs DHCP; reservation keeps .61). Tunnel is independent of nh3-pve. ⚠ While phoning home the AMT REFUSES LAN management (:16993 dark), so manage it via MeshCentral; revert body `servers/nh3-pve/amt-ethernet-static-revert.xml`. 2FA still not forced. Prime's MeshCentral login token is vaulted `pfi-tacticalrmm/meshcentral-login-token`. → `servers/pfi-tacticalrmm/README.md` diff --git a/playbooks/albok-service-host.yaml b/playbooks/albok-service-host.yaml index d245a26..33f1f59 100644 --- a/playbooks/albok-service-host.yaml +++ b/playbooks/albok-service-host.yaml @@ -20,6 +20,10 @@ steps: sudo: true shell: groupadd --gid 1511 albok-personal when: "! getent group albok-personal >/dev/null" + - name: read group albok-feedback (gid 1512) — restricted wing personal/agent-feedback + sudo: true + shell: groupadd --gid 1512 albok-feedback + when: "! getent group albok-feedback >/dev/null" - name: store root, private root and config dir (local ext4, never NFS) sudo: true shell: | @@ -32,7 +36,7 @@ steps: verify: - name: ids are the fixed numbers shell: | - test "$(getent passwd albok | cut -d: -f3,4)" = "1500:1500" && test "$(getent group albok-read | cut -d: -f3)" = 1510 && test "$(getent group albok-personal | cut -d: -f3)" = 1511 + test "$(getent passwd albok | cut -d: -f3,4)" = "1500:1500" && test "$(getent group albok-read | cut -d: -f3)" = 1510 && test "$(getent group albok-personal | cut -d: -f3)" = 1511 && test "$(getent group albok-feedback | cut -d: -f3)" = 1512 changed_when: "false" - name: roots are local (not NFS) and owned by 1500 sudo: true diff --git a/stacks/albok-service/README.md b/stacks/albok-service/README.md index 8279fcd..d03096c 100644 --- a/stacks/albok-service/README.md +++ b/stacks/albok-service/README.md @@ -6,12 +6,19 @@ The FastAPI process that owns Albok's buildings-and-wings store (`vh/albok`, - **URL:** `http://albok.nh3.internal:8392` (= `http://10.100.50.40:8392`). Container port 8390. ⚠ Host port **8392**, because **8390 on nh3-docker is the althing post office**. -- **Image:** `gitea.phasefinal.com/pfi/albok-service:0.1.1` @ `sha256:33e4e98a…` (pinned in - `compose.yaml`), built from vh/albok `e349d50` (tag albok-service/v0.1.1; albok core 0.1.1). 0.1.0 (`0b37431`) ran 2026-10-02 1754–1805. +- **Image:** `gitea.phasefinal.com/pfi/albok-service:0.1.2` @ `sha256:6f91a33f…` (pinned in + `compose.yaml`), built 2026-10-03 0250 from vh/albok `544ecfd` (tag albok-service/v0.1.2; albok core 0.1.2). + ⚠ Built from albok-dev's LOCAL commit before the operator pushed it (main was 29 ahead of origin). After the + push, check that origin's tag `albok-service/v0.1.2` is still `544ecfd`. Image labels carry the full SHA. + History: 0.1.0 (`0b37431`) 2026-10-02 1754–1805; 0.1.1 (`e349d50`) 1805 to 2026-10-03 0251. - **Health:** `GET /health` (no auth) → `"status": "ok"` on 0.1.1. (0.1.0 always said `degraded`: its canary reports `alive` and the check expected `ok`; fixed in 0.1.1.) The Docker healthcheck checks HTTP 200. - **0.1.1 also accepts numeric `default_gid` / `gid` in the config**, which would make the mounted `/etc/group` unnecessary. The current name-based config stays valid, so it is left as is. +- **Why 0.1.2:** 0.1.1 leaked a Chroma client (and its sqlite fds) on every health probe. After 9 h it held 1018 of + 1024 fds, 478 of them deleted canary files, and `/health` and `/search` answered 500. 0.1.2 closes the client. + Verified 0251: 30 `/health` calls left the fd count at 59, with 0 deleted. The same count had caught the leak on + 0.1.1. 0.1.2 also stamps `X-Albok-Service: ` on every response. ## Pieces and where they live @@ -25,8 +32,8 @@ The FastAPI process that owns Albok's buildings-and-wings store (`vh/albok`, **Identities (fixed numeric ids, created by `playbooks/albok-service-host.yaml`):** user and group `albok` 1500:1500 (the image's user), read groups `albok-read` **1510** (building `fleet`) and -`albok-personal` **1511** (building `personal`). Reserve 1512+ for restricted wings (agent-feedback, -vault) and add each to the host, `conf/group` and `group_add`. +`albok-personal` **1511** (building `personal`). **`albok-feedback` 1512** = restricted wing `personal/agent-feedback` (2026-10-03, default-deny: no viewer is +in it). Reserve 1513+ for the vault wing and any later restricted wing; add each to the host, `conf/group` and `group_add`. **Why the group file and `group_add`:** the service resolves group NAMES with `grp.getgrnam()` inside the container and `chgrp`s wing dirs as uid 1500. Without the names in the container's @@ -36,6 +43,11 @@ Verified: wings came up `drwxr-s--- albok:albok-read` / `albok:albok-personal`, **Secrets (vault):** `albok/litellm-key`, a LiteLLM key scoped to `qwen3-embedding` only (alias `albok-service`; verified 200 on embeddings, 403 on any other model); `albok/bootstrap-admin-token`, the first-start admin token (also at `/srv/albok/private/bootstrap-admin-token`, 0600). +**Nemi (albok's tagger, runs on nh3-dev):** `albok/token-nemi`, a service token minted 0255 with read+write on +`fleet/memory`, `fleet/stash` and `personal/agent-feedback`; it sees exactly those three wings (`personal/notes` hidden). +The previous token (`01M40979…`, without agent-feedback) is still live until albok-dev revokes it. +`albok/litellm-key-nemi`, alias `albok-nemi`, scoped to `gen-small`, `summarizer` and `qwen3-embedding`. Verified 200 on +all three and 403 on another model. **Backups:** nh3-docker is VM 100 on nh3-pve, in the nightly 21:00 vzdump (snapshot mode → pbs-ana), so `/srv/albok` is covered whole-VM. No file-level restic job for it. diff --git a/stacks/albok-service/compose.yaml b/stacks/albok-service/compose.yaml index 59c67fe..9fe0ef3 100644 --- a/stacks/albok-service/compose.yaml +++ b/stacks/albok-service/compose.yaml @@ -9,9 +9,10 @@ # ⚠ Exactly ONE instance per private root: a second one exits 75 on the lease. Never scale this. # ⚠ Nothing else may mount /srv/albok/store read-write. Viewers mount it :ro with group_add by gid # (1510 albok-read, 1511 albok-personal) and need GIT_OPTIONAL_LOCKS=0 + a safe.directory entry. +# 1512 albok-feedback (restricted wing personal/agent-feedback) is default-deny: no viewer gets it. services: albok-service: - image: gitea.phasefinal.com/pfi/albok-service:0.1.1@sha256:33e4e98a27412c386af60722a09e85ffe9370d63598d2aed39dd39676862d774 + image: gitea.phasefinal.com/pfi/albok-service:0.1.2@sha256:6f91a33f3e96ec7d6b043f83db2f4661e6d34034aff2c8d116b0fcec39d268be container_name: albok-service restart: unless-stopped ports: @@ -20,6 +21,7 @@ services: group_add: - "1510" - "1511" + - "1512" # albok-feedback: restricted wing personal/agent-feedback volumes: - /srv/albok/store:/srv/albok/store - /srv/albok/private:/srv/albok/private diff --git a/stacks/albok-service/conf/albok.yaml.template b/stacks/albok-service/conf/albok.yaml.template index 3398696..ab25db5 100644 --- a/stacks/albok-service/conf/albok.yaml.template +++ b/stacks/albok-service/conf/albok.yaml.template @@ -29,7 +29,7 @@ listen: port: 8390 # container port; published on the host as 8392 (8390 is the post office) # First instance, per albok-dev's ask (2026-10-02): fleet/{memory,stash}, personal/notes. -# Restricted wings (agent-feedback, vault) come later with their own groups (gids 1512+). +# 2026-10-03: + personal/agent-feedback (restricted, gid 1512). The vault wing comes later (gid 1513+). buildings: - id: fleet default_group: albok-read # host gid 1510 @@ -43,3 +43,6 @@ buildings: default_group: albok-personal # host gid 1511 wings: - id: notes + - id: agent-feedback # auto-memory (Nemi's third source); restricted, default-deny (2026-10-03) + restricted: true + group: albok-feedback # host gid 1512 diff --git a/stacks/albok-service/conf/group b/stacks/albok-service/conf/group index 4042da1..1c57c28 100644 --- a/stacks/albok-service/conf/group +++ b/stacks/albok-service/conf/group @@ -39,3 +39,4 @@ nogroup:x:65534: albok:x:1500: albok-read:x:1510:albok albok-personal:x:1511:albok +albok-feedback:x:1512:albok