167 lines
15 KiB
Markdown
167 lines
15 KiB
Markdown
# albok-service — fleet knowledgebase service (nh3-albok)
|
||
|
||
The FastAPI process that owns Albok's buildings-and-wings store (`vh/albok`,
|
||
`packages/albok-service`, contract `docs/contracts/unit2_service.contract.md`). It is the
|
||
**only writer** of the store. Deployed 2026-10-02 at albok-dev's request (operator-approved).
|
||
|
||
- **Host:** **nh3-albok** (VM 110 on nh3-pve-2, `10.100.50.44`) since 2026-10-04, a VM that runs nothing else
|
||
(`servers/nh3-albok/README.md` says why and what must never be added there). Before that: nh3-docker, 2026-10-02 to 2026-10-04.
|
||
- **URL:** `http://albok.nh3.internal:8392` (= `http://10.100.50.44:8392`). Container port 8390.
|
||
Host port **8392** is kept from nh3-docker, where 8390 was the althing post office, so no client changed.
|
||
- **Image:** `gitea.phasefinal.com/pfi/albok-service:0.1.14` @ `sha256:01b46f64…` since 2026-10-07 1804 (healthy 1804:35,
|
||
after the 1803 walk). infra-ops deployed it at albok-dev's ask (althing 8524), from a clean archive of vh/albok `7ae0d2a6`
|
||
(tag albok-service/v0.1.14). This is unit 4.3, with three additions:
|
||
- `GET .../sidecars`, a batch sidecar listing. Over 5,000 rows or 8 MiB it answers 413.
|
||
- Byte-less ingest sightings. A sighting whose stored bytes don't verify gets `needs_bytes`.
|
||
- **A browse view per wing at `<wing>/.browse/`.** It holds relative symlinks to the whole-file objects that carry no
|
||
sensitivity class, and each wing's `.git/info/exclude` keeps it out of git. Only a reindex renders the view for
|
||
objects already stored.
|
||
|
||
Detector version 2 is in this release: the URL-password rule now needs a dotted host. The first start re-scanned every
|
||
wing, and **`leaks` went from 8 to 6** (the two shutterchute placeholders cleared; fleet/memory commit `3548715`).
|
||
`/health` gains `browse_withheld`; 0 is normal, and a non-zero value is a WARN. No config change.
|
||
⚠ **`.browse/` is NOT shared with anyone yet.** The contract gates exposure on a demonstration on the chosen share:
|
||
rename `.browse/` away on the server, then try a fresh open of an unopened child from the client through a cached
|
||
listing. That open must fail. sshfs `follow_symlinks` works by path; NFS/SMB may keep directory handles. Report a
|
||
passing surface to albok-dev, who tells the operator. Rollback: `compose.yaml.pre-0.1.14` (0.1.13) is free, because no
|
||
sidecar shape changed.
|
||
Prior: 0.1.13 @ `sha256:bea95617…` (2026-10-07 1135 – 1804) (healthy 1136:17,
|
||
between walks). infra-ops deployed it at albok-dev's ask (althing 8413), from a clean archive of vh/albok `70b55ad7`
|
||
(tag albok-service/v0.1.13). Unit 4.2, the **sensitivity gate**: objects classed `adult` / `personal` / `secret` are
|
||
hidden from every reader unless the token asks to `include` them AND holds `class:<building>/<wing>/<class>` (a wing
|
||
write grant implies its classes). A rule-based detector marks credential-bearing objects `secret`. No config, key or
|
||
token change. `/health` gains `leaks` (a count; null degrades status) and `hidden_map_stale`; `GET /health/leaks`
|
||
(admin token) lists the ids. The first start scans every wing once (fleet/memory 11,355 objects in 12 s) and writes
|
||
`<private_root>/wings/*/SCANNED`. **2026-10-07 census: `leaks` 8.** Six objects hold the shared all-agents LiteLLM
|
||
key, which is distributed on purpose, so nothing needed rotating. Two (shutterchute archival-memory) are false
|
||
positives on a `http://user:…@gw` doc placeholder. Reported to albok-dev.
|
||
⚠ **Rollback is NOT free.** 0.1.12 cannot load a sidecar that has a `sensitivity` section, and the scan writes that
|
||
section and commits it per wing (fleet/memory `0032841`, personal/agent-feedback `4b7883a`; HEADs before the start
|
||
were `02012e4a` / `2342899e`). To roll back: stop the service, run the strip snippet in vh/albok
|
||
`packages/albok-service/README.md` ("Rolling back to 0.1.12") from `/srv/albok/store`, then start
|
||
`compose.yaml.pre-0.1.13`. The strip re-exposes every hidden object to every reader. To roll forward again, delete
|
||
every `/srv/albok/private/wings/*/SCANNED` first. Nemi must match the service: 0.1.4 with 0.1.13, 0.1.3 with 0.1.12.
|
||
Prior: 0.1.12 @ `sha256:a29a7e46…` (2026-10-06 1254 – 2026-10-07 1135) (healthy 1254:48,
|
||
between walks; the Nemi timer now fires at :03). infra-ops deployed it at albok-dev's ask (althing 8199), from a clean archive
|
||
of vh/albok `751a729` (tag albok-service/v0.1.12). It is the heid bug-hunt fold over unit 4.1 (contract rev 1.10). No route,
|
||
config or schema change. Rollback: `compose.yaml.pre-0.1.12` (0.1.11) stays free even after a reindex, because 0.1.11 and
|
||
0.1.12 share the generation layout. ⚠ **Since 2026-10-06 ~1530, EVERY wing is migrated** (fleet/memory g3, fleet/stash g1, personal/agent-feedback g1, personal/notes g1; the unsuffixed collections are gone), so a rollback to 0.1.10 now needs a reindex of every wing on 0.1.10 first (~23–27 min for fleet/memory). albok-dev's after-measurement under reindex: 0 mismatches in 15,176 samples, p95 1.42× idle. The 200-object-load check came in at 2.01× against a 2× bar; that ruling is Prime's.
|
||
Prior: 0.1.11 @ `sha256:4b641c3e…` (2026-10-06 0202 – 1254) (healthy 0202:23,
|
||
~20 s down between walks). Deployed by infra-ops at albok-dev's ask (althing 7991; release operator-ratified per albok-dev),
|
||
built from a clean archive of vh/albok `ffc6d4e` (tag albok-service/v0.1.11). Unit 4.1 read priority: a reindex builds a
|
||
shadow collection (`<building>.<wing>-g<N>`) and swaps it in; writes to a reindexing wing get 423 + Retry-After 60;
|
||
ingest/reindex/hard-delete run on a two-thread write pool. Config unchanged. Rollback: `compose.yaml.pre-0.1.11` brings
|
||
0.1.10 back ONLY until a wing's first 0.1.11 reindex; after that the wing's unsuffixed collection is gone, and 0.1.10
|
||
needs its own reindex (~27 min for fleet/memory) before that wing serves again.
|
||
Prior: 0.1.10 @ `sha256:7e10d852…` (2026-10-05 1022 – 2026-10-06 0202), deployed by albok-dev on
|
||
Prime's go, from vh/albok `aae3477` (tag albok-service/v0.1.10). It pins three facet-table queries to the right index
|
||
(first Nemi walk 46 s, against 6–12 min before); config unchanged. Rollback: `compose.yaml.pre-0.1.10`.
|
||
Prior: 0.1.9 @ `sha256:ba7b475b…` (0216–1022), from vh/albok `382a962` (tag albok-service/v0.1.9). It adds `embedding_write`: writes (ingest, reindex) embed on
|
||
`qwen3-embedding-bulk` (nh3-ml1 first), queries stay on `qwen3-embedding` (esh-ml1 first), and a startup vector-space
|
||
check refuses writes, never reads, on a mismatch. Unchanged sections no longer count as updated. Rollback: the
|
||
`.pre-0.1.9` copies of compose.yaml and albok.yaml beside the live files, then `compose up -d`.
|
||
Prior: 0.1.8 @ `sha256:239f900e…` (pinned in `compose.yaml` until 10-05), built
|
||
2026-10-04 ~1115 from vh/albok `e653c29` (tag albok-service/v0.1.8; heid re-pass fixes over 0.1.7: keyword-index
|
||
lock mismatch, journal-stranding replay path, bypassable 400, link-kind handling in the collapse). fleet/memory
|
||
reindexed on it per albok-dev (10,849 obj / 15,418 chunks, 1613.6 s, clean, commit 563dcae; their 0.1.7
|
||
stash + agent-feedback reindexes stand — 0.1.8 does not change what a reindex writes).
|
||
⚠ reindex path shape: POST /buildings/<building>/wings/<wing>/reindex (the wing segment is required).
|
||
Prior: 0.1.7 `sha256:0dbb8cc7…` (64f75dc), fleet/stash + personal/agent-feedback reindexed on it.
|
||
Prior: 0.1.6 `sha256:2b44c9d4…`, built
|
||
2026-10-04 ~0544 from vh/albok `6c1adcd` (tags albok-service/v0.1.6 + core v0.1.4; pushed by infra-ops on Prime's go). 0.1.6 is
|
||
the review fold on 0.1.5's search fix: the keyword-index build now takes a lock (two concurrent first queries could cross
|
||
ids or KeyError), the core search keyword is keyword-only (do not pin albok 0.1.3), and the store wrapper fails closed.
|
||
0.1.5 (below) was built 2026-10-03 ~2038 from `e70321c` (tags albok-service/v0.1.5 + core v0.1.3).
|
||
0.1.5 = search speed: an index on the facet table's object_id and one cached keyword index per wing (fleet/memory query 4.5 s → ~0.25 s warm;
|
||
the first search per wing after a start or an ingest rebuilds the keyword index, ~1.7 s). The facet index is built at the first open (~0.35 s, +17 MB).
|
||
**Measured live by albok-dev 2026-10-03 ~2245** (same idle container, before/after): fleet/memory top-5 median 3.97 s → **0.22 s** (n=5;
|
||
first query after start 2.11 s); all three wings top-20 5.62 s → **0.41 s** (n=3); an unseen query 0.39 s (n=3).
|
||
Prior: 0.1.4 (`3d6bdba`, version header from metadata) ~1611–2039; 0.1.3 (`fc4c97a`) 1558–1611; 0.1.2 (`544ecfd`) 0251–1558.
|
||
Built from albok-dev's local commit before the operator pushed it. Verified 2026-10-03 1000 with `git ls-remote`:
|
||
origin's `albok-service/v0.1.2` and `v0.1.2` are `544ecfd`, so the image matches the published tag. Image labels
|
||
carry the full SHA.
|
||
History: 0.1.0 (`0b37431`) 2026-10-02 1754–1805; 0.1.1 (`e349d50`) 1805 to 2026-10-03 0251.
|
||
- **Health:** `GET /health` (no auth) → `"status": "ok"` on 0.1.1. (0.1.0 always said `degraded`: its
|
||
canary reports `alive` and the check expected `ok`; fixed in 0.1.1.) The Docker healthcheck checks HTTP 200.
|
||
- **0.1.1 also accepts numeric `default_gid` / `gid` in the config**, which would make the mounted
|
||
`/etc/group` unnecessary. The current name-based config stays valid, so it is left as is.
|
||
- **Why 0.1.2:** 0.1.1 leaked a Chroma client (and its sqlite fds) on every health probe. After 9 h it held 1018 of
|
||
1024 fds, 478 of them deleted canary files, and `/health` and `/search` answered 500. 0.1.2 closes the client.
|
||
Verified 0251: 30 `/health` calls left the fd count at 59, with 0 deleted. The same count had caught the leak on
|
||
0.1.1. 0.1.2 also stamps `X-Albok-Service: <version>` on every response.
|
||
|
||
## Pieces and where they live
|
||
|
||
| What | Where |
|
||
|---|---|
|
||
| store root (one git repo per wing) | `/srv/albok/store` — local ext4, `albok:albok` 0711 |
|
||
| private root (lease, journal, tokens.db, per-wing chroma) | `/srv/albok/private` — local ext4, `albok:albok` 0700, **single-attach** |
|
||
| config (holds the embedder key) | `/srv/albok/etc/albok.yaml` — `root:albok` 0640; rendered from `conf/albok.yaml.template` |
|
||
| container `/etc/group` | `/opt/docker/conf/albok-service/group` (= `conf/group`) |
|
||
| compose | `/opt/docker/compose/albok-service/compose.yaml` |
|
||
|
||
**Identities (fixed numeric ids, created by `playbooks/albok-service-host.yaml`):** user and group
|
||
`albok` 1500:1500 (the image's user), read groups `albok-read` **1510** (building `fleet`) and
|
||
`albok-personal` **1511** (building `personal`). **`albok-feedback` 1512** = restricted wing `personal/agent-feedback` (2026-10-03, default-deny: no viewer is
|
||
in it). Reserve 1513+ for the vault wing and any later restricted wing; add each to the host, `conf/group` and `group_add`.
|
||
|
||
**Why the group file and `group_add`:** the service resolves group NAMES with `grp.getgrnam()`
|
||
inside the container and `chgrp`s wing dirs as uid 1500. Without the names in the container's
|
||
`/etc/group` it reports "group does not resolve", and without membership the chgrp is refused.
|
||
Verified: wings came up `drwxr-s--- albok:albok-read` / `albok:albok-personal`, drift 0.
|
||
|
||
**Secrets (vault):** `albok/litellm-key`, a LiteLLM key scoped to `qwen3-embedding` only (alias
|
||
`albok-service`; verified 200 on embeddings, 403 on any other model; widened 2026-10-05 to also cover `qwen3-embedding-bulk`, the write-path alias, c0d3384); `albok/bootstrap-admin-token`,
|
||
the first-start admin token (also at `/srv/albok/private/bootstrap-admin-token`, 0600).
|
||
**Building `library` (2026-10-09, albok-dev ask, unit 4.5 Nemi books):** wings `library/fiction` and `library/nonfiction`,
|
||
`default_group: albok-read` (1510), neither restricted, no encrypted mount; live config `albok.yaml.pre-library` kept.
|
||
Token grants are immutable, so the widening was done by minting NEW tokens: nemi `01M4HAP9G90J79HBRJB6VW6PQA` (previous six
|
||
grants plus read+write on both library wings) and albok-dev `01M4HAP9HFYZCFV76WX2HPVWT2` (`write:fleet/memory`, `read:fleet/*`,
|
||
`read:library/*`), vaulted over `albok/token-nemi` / `albok/token-albok-dev`. albok-dev swaps them into nemi.yaml and
|
||
albok-check, then revokes the previous nemi `01M40JWJ3NZ6VJ77KS8B06BCYZ` and albok-dev `01M3ZMCXE8P66MRX16MZN1D44D`.
|
||
**Nemi (albok's tagger, runs on nh3-dev):** `albok/token-nemi`, a service token minted 0255 with read+write on
|
||
`fleet/memory`, `fleet/stash` and `personal/agent-feedback`; it sees exactly those three wings (`personal/notes` hidden).
|
||
The previous token (`01M40979…`, without agent-feedback) is still live until albok-dev revokes it.
|
||
`albok/litellm-key-nemi`, alias `albok-nemi`, scoped to `gen-small`, `summarizer` and `qwen3-embedding`. Verified 200 on
|
||
all three and 403 on another model.
|
||
|
||
**Backups:** nh3-albok is VM 110 on nh3-pve-2, in that node's nightly **21:30** vzdump (snapshot mode → pbs-ana,
|
||
namespace `nh3-pve-2`; pbs-nh3's 06:30 sync mirrors it), so `/srv/albok` is covered whole-VM. No file-level restic
|
||
job. ⚠ The PBS datastores are not client-side encrypted: anyone with PBS or NAS root can read the restricted wings
|
||
out of a backup. That matters once unit 4 ingests the personal vault; unit 6's encrypted wings close it.
|
||
|
||
## Deploy / redeploy
|
||
|
||
⚠ **Do not restart the service in the middle of a Nemi walk.** The hourly walk on nh3-dev (`services/albok-nemi/`) fails with
|
||
`Unreachable` and mails OnFailure; that happened 2026-10-03 16:11 at the 0.1.4 redeploy. It is harmless, since the next walk redoes the
|
||
work and the service commits per batch, but avoidable. Check first, on nh3-dev as lkraven:
|
||
`flock -n ~/.local/state/albok-nemi/run.lock true && echo idle || echo WALK RUNNING`
|
||
|
||
```bash
|
||
# 1. host prep (idempotent)
|
||
scripts/elway infra-ops@10.100.50.44 --playbook playbooks/albok-service-host.yaml
|
||
# 2. config: render the template with the vaulted key (never commit the rendered file)
|
||
umask 077; secret get albok/litellm-key | tr -d '\n' | python3 -c "import sys; k=sys.stdin.read(); \
|
||
open('/tmp/albok.yaml','w').write(open('stacks/albok-service/conf/albok.yaml.template').read().replace('__ALBOK_LITELLM_KEY__', k))"
|
||
scripts/elway infra-ops@10.100.50.44 --upload /tmp/albok.yaml:/srv/albok/etc/albok.yaml:0640 --sudo --owner root:albok
|
||
# then delete /tmp/albok.yaml by its literal path
|
||
# 3. compose + conf, then start (root holds the registry login on nh3-albok: claude-bot, vaulted nh3-dev/.config/claude-bot/gitea-token)
|
||
scripts/deploy-stack.sh nh3-albok albok-service
|
||
ssh infra-ops@10.100.50.44 'cd /opt/docker/compose/albok-service && sudo docker compose up -d'
|
||
```
|
||
|
||
**Image rebuild** (from a clean archive, never a working tree):
|
||
|
||
```bash
|
||
git -C ~/development/albok archive <sha> | tar -x -C <scratch>
|
||
cd <scratch> && docker build -f packages/albok-service/Dockerfile -t gitea.phasefinal.com/pfi/albok-service:<ver> .
|
||
secret get nh3-dev/.config/claude-bot/gitea-token-sdkops | docker login gitea.phasefinal.com -u claude-bot --password-stdin
|
||
docker push gitea.phasefinal.com/pfi/albok-service:<ver> # then pin the new digest in compose.yaml
|
||
```
|
||
|
||
## Viewers (read-only access to the store)
|
||
|
||
Mount `/srv/albok/store` **`:ro`**, `group_add` the read group's **numeric** gid (1510 / 1511), set
|
||
`GIT_OPTIONAL_LOCKS=0`, and add `safe.directory` for the path as the viewer sees it. Nothing else
|
||
may mount the store read-write.
|