vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5cb610b43389f34f6bef585933c4c48f6e6fd8f8
esh-pfi-infrastructure/stacks/albok-service
T
History

README.md

albok-service — fleet knowledgebase service (nh3-albok)

The FastAPI process that owns Albok's buildings-and-wings store (vh/albok, packages/albok-service, contract docs/contracts/unit2_service.contract.md). It is the only writer of the store. Deployed 2026-10-02 at albok-dev's request (operator-approved).

  • Host: nh3-albok (VM 110 on nh3-pve-2, 10.100.50.44) since 2026-10-04, a VM that runs nothing else (servers/nh3-albok/README.md says why and what must never be added there). Before that: nh3-docker, 2026-10-02 to 2026-10-04.

  • URL: http://albok.nh3.internal:8392 (= http://10.100.50.44:8392). Container port 8390. Host port 8392 is kept from nh3-docker, where 8390 was the althing post office, so no client changed.

  • Image: gitea.phasefinal.com/pfi/albok-service:0.1.14 @ sha256:01b46f64… since 2026-10-07 1804 (healthy 1804:35, after the 1803 walk). infra-ops deployed it at albok-dev's ask (althing 8524), from a clean archive of vh/albok 7ae0d2a6 (tag albok-service/v0.1.14). This is unit 4.3, with three additions:

    • GET .../sidecars, a batch sidecar listing. Over 5,000 rows or 8 MiB it answers 413.
    • Byte-less ingest sightings. A sighting whose stored bytes don't verify gets needs_bytes.
    • A browse view per wing at <wing>/.browse/. It holds relative symlinks to the whole-file objects that carry no sensitivity class, and each wing's .git/info/exclude keeps it out of git. Only a reindex renders the view for objects already stored.

    Detector version 2 is in this release: the URL-password rule now needs a dotted host. The first start re-scanned every wing, and leaks went from 8 to 6 (the two shutterchute placeholders cleared; fleet/memory commit 3548715). /health gains browse_withheld; 0 is normal, and a non-zero value is a WARN. No config change. ⚠ .browse/ is NOT shared with anyone yet. The contract gates exposure on a demonstration on the chosen share: rename .browse/ away on the server, then try a fresh open of an unopened child from the client through a cached listing. That open must fail. sshfs follow_symlinks works by path; NFS/SMB may keep directory handles. Report a passing surface to albok-dev, who tells the operator. Rollback: compose.yaml.pre-0.1.14 (0.1.13) is free, because no sidecar shape changed. Prior: 0.1.13 @ sha256:bea95617… (2026-10-07 1135 – 1804) (healthy 1136:17, between walks). infra-ops deployed it at albok-dev's ask (althing 8413), from a clean archive of vh/albok 70b55ad7 (tag albok-service/v0.1.13). Unit 4.2, the sensitivity gate: objects classed adult / personal / secret are hidden from every reader unless the token asks to include them AND holds class:<building>/<wing>/<class> (a wing write grant implies its classes). A rule-based detector marks credential-bearing objects secret. No config, key or token change. /health gains leaks (a count; null degrades status) and hidden_map_stale; GET /health/leaks (admin token) lists the ids. The first start scans every wing once (fleet/memory 11,355 objects in 12 s) and writes <private_root>/wings/*/SCANNED. 2026-10-07 census: leaks 8. Six objects hold the shared all-agents LiteLLM key, which is distributed on purpose, so nothing needed rotating. Two (shutterchute archival-memory) are false positives on a http://user:…@gw doc placeholder. Reported to albok-dev. ⚠ Rollback is NOT free. 0.1.12 cannot load a sidecar that has a sensitivity section, and the scan writes that section and commits it per wing (fleet/memory 0032841, personal/agent-feedback 4b7883a; HEADs before the start were 02012e4a / 2342899e). To roll back: stop the service, run the strip snippet in vh/albok packages/albok-service/README.md ("Rolling back to 0.1.12") from /srv/albok/store, then start compose.yaml.pre-0.1.13. The strip re-exposes every hidden object to every reader. To roll forward again, delete every /srv/albok/private/wings/*/SCANNED first. Nemi must match the service: 0.1.4 with 0.1.13, 0.1.3 with 0.1.12. Prior: 0.1.12 @ sha256:a29a7e46… (2026-10-06 1254 – 2026-10-07 1135) (healthy 1254:48, between walks; the Nemi timer now fires at :03). infra-ops deployed it at albok-dev's ask (althing 8199), from a clean archive of vh/albok 751a729 (tag albok-service/v0.1.12). It is the heid bug-hunt fold over unit 4.1 (contract rev 1.10). No route, config or schema change. Rollback: compose.yaml.pre-0.1.12 (0.1.11) stays free even after a reindex, because 0.1.11 and 0.1.12 share the generation layout. ⚠ Since 2026-10-06 ~1530, EVERY wing is migrated (fleet/memory g3, fleet/stash g1, personal/agent-feedback g1, personal/notes g1; the unsuffixed collections are gone), so a rollback to 0.1.10 now needs a reindex of every wing on 0.1.10 first (~23–27 min for fleet/memory). albok-dev's after-measurement under reindex: 0 mismatches in 15,176 samples, p95 1.42× idle. The 200-object-load check came in at 2.01× against a 2× bar; that ruling is Prime's. Prior: 0.1.11 @ sha256:4b641c3e… (2026-10-06 0202 – 1254) (healthy 0202:23, ~20 s down between walks). Deployed by infra-ops at albok-dev's ask (althing 7991; release operator-ratified per albok-dev), built from a clean archive of vh/albok ffc6d4e (tag albok-service/v0.1.11). Unit 4.1 read priority: a reindex builds a shadow collection (<building>.<wing>-g<N>) and swaps it in; writes to a reindexing wing get 423 + Retry-After 60; ingest/reindex/hard-delete run on a two-thread write pool. Config unchanged. Rollback: compose.yaml.pre-0.1.11 brings 0.1.10 back ONLY until a wing's first 0.1.11 reindex; after that the wing's unsuffixed collection is gone, and 0.1.10 needs its own reindex (~27 min for fleet/memory) before that wing serves again. Prior: 0.1.10 @ sha256:7e10d852… (2026-10-05 1022 – 2026-10-06 0202), deployed by albok-dev on Prime's go, from vh/albok aae3477 (tag albok-service/v0.1.10). It pins three facet-table queries to the right index (first Nemi walk 46 s, against 6–12 min before); config unchanged. Rollback: compose.yaml.pre-0.1.10. Prior: 0.1.9 @ sha256:ba7b475b… (0216–1022), from vh/albok 382a962 (tag albok-service/v0.1.9). It adds embedding_write: writes (ingest, reindex) embed on qwen3-embedding-bulk (nh3-ml1 first), queries stay on qwen3-embedding (esh-ml1 first), and a startup vector-space check refuses writes, never reads, on a mismatch. Unchanged sections no longer count as updated. Rollback: the .pre-0.1.9 copies of compose.yaml and albok.yaml beside the live files, then compose up -d. Prior: 0.1.8 @ sha256:239f900e… (pinned in compose.yaml until 10-05), built 2026-10-04 ~1115 from vh/albok e653c29 (tag albok-service/v0.1.8; heid re-pass fixes over 0.1.7: keyword-index lock mismatch, journal-stranding replay path, bypassable 400, link-kind handling in the collapse). fleet/memory reindexed on it per albok-dev (10,849 obj / 15,418 chunks, 1613.6 s, clean, commit 563dcae; their 0.1.7 stash + agent-feedback reindexes stand — 0.1.8 does not change what a reindex writes). ⚠ reindex path shape: POST /buildings//wings//reindex (the wing segment is required). Prior: 0.1.7 sha256:0dbb8cc7… (64f75dc), fleet/stash + personal/agent-feedback reindexed on it. Prior: 0.1.6 sha256:2b44c9d4…, built 2026-10-04 ~0544 from vh/albok 6c1adcd (tags albok-service/v0.1.6 + core v0.1.4; pushed by infra-ops on Prime's go). 0.1.6 is the review fold on 0.1.5's search fix: the keyword-index build now takes a lock (two concurrent first queries could cross ids or KeyError), the core search keyword is keyword-only (do not pin albok 0.1.3), and the store wrapper fails closed. 0.1.5 (below) was built 2026-10-03 ~2038 from e70321c (tags albok-service/v0.1.5 + core v0.1.3). 0.1.5 = search speed: an index on the facet table's object_id and one cached keyword index per wing (fleet/memory query 4.5 s → ~0.25 s warm; the first search per wing after a start or an ingest rebuilds the keyword index, ~1.7 s). The facet index is built at the first open (~0.35 s, +17 MB). Measured live by albok-dev 2026-10-03 ~2245 (same idle container, before/after): fleet/memory top-5 median 3.97 s → 0.22 s (n=5; first query after start 2.11 s); all three wings top-20 5.62 s → 0.41 s (n=3); an unseen query 0.39 s (n=3). Prior: 0.1.4 (3d6bdba, version header from metadata) ~1611–2039; 0.1.3 (fc4c97a) 1558–1611; 0.1.2 (544ecfd) 0251–1558. Built from albok-dev's local commit before the operator pushed it. Verified 2026-10-03 1000 with git ls-remote: origin's albok-service/v0.1.2 and v0.1.2 are 544ecfd, so the image matches the published tag. Image labels carry the full SHA. History: 0.1.0 (0b37431) 2026-10-02 1754–1805; 0.1.1 (e349d50) 1805 to 2026-10-03 0251.

  • Health: GET /health (no auth) → "status": "ok" on 0.1.1. (0.1.0 always said degraded: its canary reports alive and the check expected ok; fixed in 0.1.1.) The Docker healthcheck checks HTTP 200.

  • 0.1.1 also accepts numeric default_gid / gid in the config, which would make the mounted /etc/group unnecessary. The current name-based config stays valid, so it is left as is.

  • Why 0.1.2: 0.1.1 leaked a Chroma client (and its sqlite fds) on every health probe. After 9 h it held 1018 of 1024 fds, 478 of them deleted canary files, and /health and /search answered 500. 0.1.2 closes the client. Verified 0251: 30 /health calls left the fd count at 59, with 0 deleted. The same count had caught the leak on 0.1.1. 0.1.2 also stamps X-Albok-Service: <version> on every response.

Pieces and where they live

What Where
store root (one git repo per wing) /srv/albok/store — local ext4, albok:albok 0711
private root (lease, journal, tokens.db, per-wing chroma) /srv/albok/private — local ext4, albok:albok 0700, single-attach
config (holds the embedder key) /srv/albok/etc/albok.yaml — root:albok 0640; rendered from conf/albok.yaml.template
container /etc/group /opt/docker/conf/albok-service/group (= conf/group)
compose /opt/docker/compose/albok-service/compose.yaml

Identities (fixed numeric ids, created by playbooks/albok-service-host.yaml): user and group albok 1500:1500 (the image's user), read groups albok-read 1510 (building fleet) and albok-personal 1511 (building personal). albok-feedback 1512 = restricted wing personal/agent-feedback (2026-10-03, default-deny: no viewer is in it). Reserve 1513+ for the vault wing and any later restricted wing; add each to the host, conf/group and group_add.

Why the group file and group_add: the service resolves group NAMES with grp.getgrnam() inside the container and chgrps wing dirs as uid 1500. Without the names in the container's /etc/group it reports "group does not resolve", and without membership the chgrp is refused. Verified: wings came up drwxr-s--- albok:albok-read / albok:albok-personal, drift 0.

Secrets (vault): albok/litellm-key, a LiteLLM key scoped to qwen3-embedding only (alias albok-service; verified 200 on embeddings, 403 on any other model; widened 2026-10-05 to also cover qwen3-embedding-bulk, the write-path alias, c0d3384); albok/bootstrap-admin-token, the first-start admin token (also at /srv/albok/private/bootstrap-admin-token, 0600). Building library (2026-10-09, albok-dev ask, unit 4.5 Nemi books): wings library/fiction and library/nonfiction, default_group: albok-read (1510), neither restricted, no encrypted mount; live config albok.yaml.pre-library kept. Token grants are immutable, so the widening was done by minting NEW tokens: nemi 01M4HAP9G90J79HBRJB6VW6PQA (previous six grants plus read+write on both library wings) and albok-dev 01M4HAP9HFYZCFV76WX2HPVWT2 (write:fleet/memory, read:fleet/*, read:library/*), vaulted over albok/token-nemi / albok/token-albok-dev. albok-dev swaps them into nemi.yaml and albok-check (swapped 1459, proven by the 1503 walk) and revoked the previous nemi 01M40JWJ3NZ6VJ77KS8B06BCYZ and albok-dev 01M3ZMCXE8P66MRX16MZN1D44D at 1555 10-09. Live set: bootstrap-admin, svos-knowledge-nh3-svos, the new nemi, the new albok-dev. Nemi (albok's tagger, runs on nh3-dev): albok/token-nemi, a service token minted 0255 with read+write on fleet/memory, fleet/stash and personal/agent-feedback; it sees exactly those three wings (personal/notes hidden). The previous token (01M40979…, without agent-feedback) is still live until albok-dev revokes it. albok/litellm-key-nemi, alias albok-nemi, scoped to gen-small, summarizer and qwen3-embedding. Verified 200 on all three and 403 on another model.

Backups: nh3-albok is VM 110 on nh3-pve-2, in that node's nightly 21:30 vzdump (snapshot mode → pbs-ana, namespace nh3-pve-2; pbs-nh3's 06:30 sync mirrors it), so /srv/albok is covered whole-VM. No file-level restic job. ⚠ The PBS datastores are not client-side encrypted: anyone with PBS or NAS root can read the restricted wings out of a backup. That matters once unit 4 ingests the personal vault; unit 6's encrypted wings close it.

Deploy / redeploy

⚠ Do not restart the service in the middle of a Nemi walk. The hourly walk on nh3-dev (services/albok-nemi/) fails with Unreachable and mails OnFailure; that happened 2026-10-03 16:11 at the 0.1.4 redeploy. It is harmless, since the next walk redoes the work and the service commits per batch, but avoidable. Check first, on nh3-dev as lkraven: flock -n ~/.local/state/albok-nemi/run.lock true && echo idle || echo WALK RUNNING

# 1. host prep (idempotent)
scripts/elway infra-ops@10.100.50.44 --playbook playbooks/albok-service-host.yaml
# 2. config: render the template with the vaulted key (never commit the rendered file)
umask 077; secret get albok/litellm-key | tr -d '\n' | python3 -c "import sys; k=sys.stdin.read(); \
  open('/tmp/albok.yaml','w').write(open('stacks/albok-service/conf/albok.yaml.template').read().replace('__ALBOK_LITELLM_KEY__', k))"
scripts/elway infra-ops@10.100.50.44 --upload /tmp/albok.yaml:/srv/albok/etc/albok.yaml:0640 --sudo --owner root:albok
# then delete /tmp/albok.yaml by its literal path
# 3. compose + conf, then start (root holds the registry login on nh3-albok: claude-bot, vaulted nh3-dev/.config/claude-bot/gitea-token)
scripts/deploy-stack.sh nh3-albok albok-service
ssh infra-ops@10.100.50.44 'cd /opt/docker/compose/albok-service && sudo docker compose up -d'

Image rebuild (from a clean archive, never a working tree):

git -C ~/development/albok archive <sha> | tar -x -C <scratch>
cd <scratch> && docker build -f packages/albok-service/Dockerfile -t gitea.phasefinal.com/pfi/albok-service:<ver> .
secret get nh3-dev/.config/claude-bot/gitea-token-sdkops | docker login gitea.phasefinal.com -u claude-bot --password-stdin
docker push gitea.phasefinal.com/pfi/albok-service:<ver>     # then pin the new digest in compose.yaml

Viewers (read-only access to the store)

Mount /srv/albok/store :ro, group_add the read group's numeric gid (1510 / 1511), set GIT_OPTIONAL_LOCKS=0, and add safe.directory for the path as the viewer sees it. Nothing else may mount the store read-write.

Reference in New Issue View Git Blame Copy Permalink