esh-pfi-infrastructure/stacks/albok-service/README.md

albok-service — fleet knowledgebase service (nh3-albok)

The FastAPI process that owns Albok's buildings-and-wings store (vh/albok, packages/albok-service, contract docs/contracts/unit2_service.contract.md). It is the only writer of the store. Deployed 2026-10-02 at albok-dev's request (operator-approved).

• Host: nh3-albok (VM 110 on nh3-pve-2, 10.100.50.44) since 2026-10-04, a VM that runs nothing else (servers/nh3-albok/README.md says why and what must never be added there). Before that: nh3-docker, 2026-10-02 to 2026-10-04.
• URL: http://albok.nh3.internal:8392 (= http://10.100.50.44:8392). Container port 8390. Host port 8392 is kept from nh3-docker, where 8390 was the althing post office, so no client changed.
• Image: gitea.phasefinal.com/pfi/albok-service:0.1.12 @ sha256:a29a7e46… since 2026-10-06 1254 (healthy 1254:48, between walks; the Nemi timer now fires at :03). infra-ops deployed it at albok-dev's ask (althing 8199), from a clean archive of vh/albok 751a729 (tag albok-service/v0.1.12). It is the heid bug-hunt fold over unit 4.1 (contract rev 1.10). No route, config or schema change. Rollback: compose.yaml.pre-0.1.12 (0.1.11) stays free even after a reindex, because 0.1.11 and 0.1.12 share the generation layout. The 0.1.10 caveat below still applies. Prior: 0.1.11 @ sha256:4b641c3e… (2026-10-06 0202 – 1254) (healthy 0202:23, ~20 s down between walks). Deployed by infra-ops at albok-dev's ask (althing 7991; release operator-ratified per albok-dev), built from a clean archive of vh/albok ffc6d4e (tag albok-service/v0.1.11). Unit 4.1 read priority: a reindex builds a shadow collection (<building>.<wing>-g<N>) and swaps it in; writes to a reindexing wing get 423 + Retry-After 60; ingest/reindex/hard-delete run on a two-thread write pool. Config unchanged. Rollback: compose.yaml.pre-0.1.11 brings 0.1.10 back ONLY until a wing's first 0.1.11 reindex; after that the wing's unsuffixed collection is gone, and 0.1.10 needs its own reindex (~27 min for fleet/memory) before that wing serves again. Prior: 0.1.10 @ sha256:7e10d852… (2026-10-05 1022 – 2026-10-06 0202), deployed by albok-dev on Prime's go, from vh/albok aae3477 (tag albok-service/v0.1.10). It pins three facet-table queries to the right index (first Nemi walk 46 s, against 6–12 min before); config unchanged. Rollback: compose.yaml.pre-0.1.10. Prior: 0.1.9 @ sha256:ba7b475b… (0216–1022), from vh/albok 382a962 (tag albok-service/v0.1.9). It adds embedding_write: writes (ingest, reindex) embed on qwen3-embedding-bulk (nh3-ml1 first), queries stay on qwen3-embedding (esh-ml1 first), and a startup vector-space check refuses writes, never reads, on a mismatch. Unchanged sections no longer count as updated. Rollback: the .pre-0.1.9 copies of compose.yaml and albok.yaml beside the live files, then compose up -d. Prior: 0.1.8 @ sha256:239f900e… (pinned in compose.yaml until 10-05), built 2026-10-04 ~1115 from vh/albok e653c29 (tag albok-service/v0.1.8; heid re-pass fixes over 0.1.7: keyword-index lock mismatch, journal-stranding replay path, bypassable 400, link-kind handling in the collapse). fleet/memory reindexed on it per albok-dev (10,849 obj / 15,418 chunks, 1613.6 s, clean, commit 563dcae; their 0.1.7 stash + agent-feedback reindexes stand — 0.1.8 does not change what a reindex writes). ⚠ reindex path shape: POST /buildings//wings//reindex (the wing segment is required). Prior: 0.1.7 sha256:0dbb8cc7… (64f75dc), fleet/stash + personal/agent-feedback reindexed on it. Prior: 0.1.6 sha256:2b44c9d4…, built 2026-10-04 ~0544 from vh/albok 6c1adcd (tags albok-service/v0.1.6 + core v0.1.4; pushed by infra-ops on Prime's go). 0.1.6 is the review fold on 0.1.5's search fix: the keyword-index build now takes a lock (two concurrent first queries could cross ids or KeyError), the core search keyword is keyword-only (do not pin albok 0.1.3), and the store wrapper fails closed. 0.1.5 (below) was built 2026-10-03 ~2038 from e70321c (tags albok-service/v0.1.5 + core v0.1.3). 0.1.5 = search speed: an index on the facet table's object_id and one cached keyword index per wing (fleet/memory query 4.5 s → ~0.25 s warm; the first search per wing after a start or an ingest rebuilds the keyword index, ~1.7 s). The facet index is built at the first open (~0.35 s, +17 MB). Measured live by albok-dev 2026-10-03 ~2245 (same idle container, before/after): fleet/memory top-5 median 3.97 s → 0.22 s (n=5; first query after start 2.11 s); all three wings top-20 5.62 s → 0.41 s (n=3); an unseen query 0.39 s (n=3). Prior: 0.1.4 (3d6bdba, version header from metadata) ~1611–2039; 0.1.3 (fc4c97a) 1558–1611; 0.1.2 (544ecfd) 0251–1558. Built from albok-dev's local commit before the operator pushed it. Verified 2026-10-03 1000 with git ls-remote: origin's albok-service/v0.1.2 and v0.1.2 are 544ecfd, so the image matches the published tag. Image labels carry the full SHA. History: 0.1.0 (0b37431) 2026-10-02 1754–1805; 0.1.1 (e349d50) 1805 to 2026-10-03 0251.
• Health: GET /health (no auth) → "status": "ok" on 0.1.1. (0.1.0 always said degraded: its canary reports alive and the check expected ok; fixed in 0.1.1.) The Docker healthcheck checks HTTP 200.
• 0.1.1 also accepts numeric default_gid / gid in the config, which would make the mounted /etc/group unnecessary. The current name-based config stays valid, so it is left as is.
• Why 0.1.2: 0.1.1 leaked a Chroma client (and its sqlite fds) on every health probe. After 9 h it held 1018 of 1024 fds, 478 of them deleted canary files, and /health and /search answered 500. 0.1.2 closes the client. Verified 0251: 30 /health calls left the fd count at 59, with 0 deleted. The same count had caught the leak on 0.1.1. 0.1.2 also stamps X-Albok-Service: <version> on every response.

Pieces and where they live

What	Where
store root (one git repo per wing)	/srv/albok/store — local ext4, albok:albok 0711
private root (lease, journal, tokens.db, per-wing chroma)	/srv/albok/private — local ext4, albok:albok 0700, single-attach
config (holds the embedder key)	/srv/albok/etc/albok.yaml — root:albok 0640; rendered from conf/albok.yaml.template
container /etc/group	/opt/docker/conf/albok-service/group (= conf/group)
compose	/opt/docker/compose/albok-service/compose.yaml

Identities (fixed numeric ids, created by playbooks/albok-service-host.yaml): user and group albok 1500:1500 (the image's user), read groups albok-read 1510 (building fleet) and albok-personal 1511 (building personal). albok-feedback 1512 = restricted wing personal/agent-feedback (2026-10-03, default-deny: no viewer is in it). Reserve 1513+ for the vault wing and any later restricted wing; add each to the host, conf/group and group_add.

Why the group file and group_add: the service resolves group NAMES with grp.getgrnam() inside the container and chgrps wing dirs as uid 1500. Without the names in the container's /etc/group it reports "group does not resolve", and without membership the chgrp is refused. Verified: wings came up drwxr-s--- albok:albok-read / albok:albok-personal, drift 0.

Secrets (vault): albok/litellm-key, a LiteLLM key scoped to qwen3-embedding only (alias albok-service; verified 200 on embeddings, 403 on any other model; widened 2026-10-05 to also cover qwen3-embedding-bulk, the write-path alias, c0d3384); albok/bootstrap-admin-token, the first-start admin token (also at /srv/albok/private/bootstrap-admin-token, 0600). Nemi (albok's tagger, runs on nh3-dev): albok/token-nemi, a service token minted 0255 with read+write on fleet/memory, fleet/stash and personal/agent-feedback; it sees exactly those three wings (personal/notes hidden). The previous token (01M40979…, without agent-feedback) is still live until albok-dev revokes it. albok/litellm-key-nemi, alias albok-nemi, scoped to gen-small, summarizer and qwen3-embedding. Verified 200 on all three and 403 on another model.

Backups: nh3-albok is VM 110 on nh3-pve-2, in that node's nightly 21:30 vzdump (snapshot mode → pbs-ana, namespace nh3-pve-2; pbs-nh3's 06:30 sync mirrors it), so /srv/albok is covered whole-VM. No file-level restic job. ⚠ The PBS datastores are not client-side encrypted: anyone with PBS or NAS root can read the restricted wings out of a backup. That matters once unit 4 ingests the personal vault; unit 6's encrypted wings close it.

Deploy / redeploy

⚠ Do not restart the service in the middle of a Nemi walk. The hourly walk on nh3-dev (services/albok-nemi/) fails with Unreachable and mails OnFailure; that happened 2026-10-03 16:11 at the 0.1.4 redeploy. It is harmless, since the next walk redoes the work and the service commits per batch, but avoidable. Check first, on nh3-dev as lkraven: flock -n ~/.local/state/albok-nemi/run.lock true && echo idle || echo WALK RUNNING

# 1. host prep (idempotent)
scripts/elway infra-ops@10.100.50.44 --playbook playbooks/albok-service-host.yaml
# 2. config: render the template with the vaulted key (never commit the rendered file)
umask 077; secret get albok/litellm-key | tr -d '\n' | python3 -c "import sys; k=sys.stdin.read(); \
  open('/tmp/albok.yaml','w').write(open('stacks/albok-service/conf/albok.yaml.template').read().replace('__ALBOK_LITELLM_KEY__', k))"
scripts/elway infra-ops@10.100.50.44 --upload /tmp/albok.yaml:/srv/albok/etc/albok.yaml:0640 --sudo --owner root:albok
# then delete /tmp/albok.yaml by its literal path
# 3. compose + conf, then start (root holds the registry login on nh3-albok: claude-bot, vaulted nh3-dev/.config/claude-bot/gitea-token)
scripts/deploy-stack.sh nh3-albok albok-service
ssh infra-ops@10.100.50.44 'cd /opt/docker/compose/albok-service && sudo docker compose up -d'

Image rebuild (from a clean archive, never a working tree):

git -C ~/development/albok archive <sha> | tar -x -C <scratch>
cd <scratch> && docker build -f packages/albok-service/Dockerfile -t gitea.phasefinal.com/pfi/albok-service:<ver> .
secret get nh3-dev/.config/claude-bot/gitea-token-sdkops | docker login gitea.phasefinal.com -u claude-bot --password-stdin
docker push gitea.phasefinal.com/pfi/albok-service:<ver>     # then pin the new digest in compose.yaml

Viewers (read-only access to the store)

Mount /srv/albok/store :ro, group_add the read group's numeric gid (1510 / 1511), set GIT_OPTIONAL_LOCKS=0, and add safe.directory for the path as the viewer sees it. Nothing else may mount the store read-write.