vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(albok): move albok-service to a dedicated VM, nh3-albok (VM 110 on nh3-pve-2)

Browse Source 
albok-service leaves nh3-docker before unit 4 puts the personal vault and
licensed books into restricted wings. On nh3-docker, portainer, portainer_agent,
dockge (socket rw) and dockerd on plaintext tcp/2375 were each root over
/srv/albok. The new host runs only albok-service: no socket containers and no
TCP Docker API, so its Homepage tile is a static services.yaml entry.

- nh3-pve-2: vmbr0 made VLAN-aware (applied by reboot, no guests running);
  pbs-ana storage in the new namespace nh3-pve-2, plus a 21:30 vzdump job
  (pbs-nh3's recursive sync mirrors it)
- VM 110 nh3-albok, 10.100.50.44 on VLAN 50, Debian 13 cloud-init, 4 vCPU
  x86-64-v3 / 12 GiB / 100 GB; built to fleet conventions (vh 1000,
  infra-ops 850, docker 851)
- compose: mem_limit 4g -> 8g, homepage labels removed; host port 8392 kept
- cutover by tar stream, verified by digest (contents plus owner, group, mode,
  mtime and type identical); albok.nh3.internal now points at nh3-albok
- old /srv/albok stays on nh3-docker (container removed, compose renamed)
  until acceptance and the operator's approval to delete it
- sudo-io archive hosts and ops-log STACK_HOSTS include nh3-albok
This commit is contained in:
vh
2026-10-04 23:33:24 -07:00
parent 867f33d672
commit d3f59aa2c5
17 changed files with 607 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
servers/nh3-albok/README.md
+57
View File
@@ -0,0 +1,57 @@
# nh3-albok: the albok-service host (VM 110 on nh3-pve-2, NH3)
A VM that runs **only** `albok-service`, the fleet knowledgebase (`stacks/albok-service/`). Built 2026-10-04 at
albok-dev's request, relaying Prime (~2235): albok moved off nh3-docker before unit 4 puts the operator's personal
vault and licensed books into restricted wings.
**Why it has its own VM.** On nh3-docker, four things were root over `/srv/albok`, restricted wings included:
portainer, portainer_agent and dockge (docker socket read-write), and dockerd itself on plaintext **tcp/2375**
for Homepage discovery, reachable from the whole LAN. Memory was the second reason: albok sat at 2.4 of a 4 GiB
cap on a 7.7 GiB host shared with the post office and AdGuard, so a reindex under pressure could take DNS or the
bus with it. Unit 6 (encrypted wings keyed from the vault) also wants a host whose only tenant is albok.
| | |
|---|---|
| Address | `10.100.50.44`, VLAN 50 (nh3-servers), static via cloud-init; `nh3-albok.nh3.internal`. Service alias `albok.nh3.internal` points here |
| Guest | VM **110** on **nh3-pve-2** (first guest there). `onboot 1`, tag `albok` |
| Shape | 4 vCPU `x86-64-v3` (AVX2; still restorable on nh3-pve's i9-13900H), 12 GiB, 100 GB `vmstore` (LVM-thin, discard on), virtio-scsi-single + iothread |
| OS | Debian 13 genericcloud + cloud-init; Docker CE 29.8 from Docker's repo |
| Identities | `vh` 1000 (operator keys), `infra-ops` 850 (NOPASSWD sudo, `log_output`, BULK_DATA exempt), `docker` gid 851; albok ids 1500/1510/1511/1512 (see the stack README) |
| Service | `albok-service` on host **8392** → container 8390, `mem_limit 8g`. `/srv/albok/{store,private,etc}` on the root ext4 |
| Backups | nh3-pve-2 vzdump job, nightly **2130**, snapshot mode → **pbs-ana**, namespace **`nh3-pve-2`**; pbs-nh3's 0630 sync mirrors it |
## ⚠ Keep it single-tenant
The point of this host is that nothing else can reach the store as root. So:
- **No** portainer, dockge, beszel-agent-with-socket or any container that mounts `/var/run/docker.sock`.
- **No** `-H tcp://` on dockerd. Homepage therefore cannot discover it; the Albok tile is a **static** entry in
`stacks/homepage/conf/services.yaml`. Do not add this host to `stacks/homepage/conf/docker.yaml`.
- **Nothing else mounts `/srv/albok/store` read-write.** Viewers follow the rules in the stack README.
- Who is root here anyway: anyone with root on **nh3-pve-2** (reads the disk), the **PBS** datastores (backups are
not client-side encrypted), and the `vh` / `infra-ops` sudoers. That circle is the residual exposure.
## Build (reproducible)
```bash
scripts/elway infra-ops@10.100.250.62 --playbook playbooks/nh3-pve-2-vmbr0-vlan-aware.yaml # once, then reboot nh3-pve-2
scripts/elway infra-ops@10.100.250.62 --playbook playbooks/nh3-albok-vm.yaml # VM 110 + cloud-init
scripts/elway vh@10.100.50.44 --playbook playbooks/bootstrap-infra-ops-user.yaml # infra-ops 850, as vh
scripts/elway infra-ops@10.100.50.44 --playbook playbooks/nh3-albok-host.yaml # docker, tree, takes the bootstrap key off vh
scripts/elway infra-ops@10.100.50.44 --playbook playbooks/albok-service-host.yaml # albok ids + /srv/albok
# then the stack README's deploy steps (config, deploy-stack.sh nh3-albok albok-service, registry login, up -d)
```
Registry: root is logged in to `gitea.phasefinal.com` as `claude-bot` with the vaulted
`nh3-dev/.config/claude-bot/gitea-token`, the same credential nh3-docker's root uses (matched by hash, 2026-10-04).
## Move from nh3-docker (2026-10-04)
The store, private root and config moved as one `tar` stream (`--format=posix --numeric-owner --xattrs --acls`,
both ends under `sudo /usr/bin/tar`, which the BULK_DATA rule keeps out of the sudo-io recordings). It was checked
by a manifest digest (path, uid, gid, mode, size, mtime, type) and a content-hash digest compared on both ends.
Only digests were printed, so the restricted wings' paths never reached a terminal or a recording.
On nh3-docker the container was removed and `compose.yaml` renamed to
`compose.yaml.moved-to-nh3-albok-2026-10-04`, so a stray `compose up` there cannot start a second writer on a stale
copy. **`/srv/albok` is still on nh3-docker** until Prime approves deleting it after acceptance.
servers/nh3-albok/ssh-target
+1
View File
@@ -0,0 +1 @@
infra-ops@10.100.50.44
servers/nh3-albok/system-details.txt
+189
View File
File diff suppressed because one or more lines are too long
servers/nh3-pve-2/README.md
+25 -5
View File
@@ -1,12 +1,28 @@
# nh3-pve-2: Minisforum MS-03, Proxmox VE 9 (NH3)
The NH3 MS-03, twin of esh-pve-2. **Purpose is TBD on purpose** (Prime, 2026-10-02): a high-powered PVE host,
possibly a dev environment for security-software work. Standalone node, not clustered with nh3-pve.
possibly a dev environment for security-software work. First guest: nh3-albok (2026-10-04, below). Standalone node, not clustered with nh3-pve.
- **Host:** `10.100.250.62` on nh3-mgmt (`nh3-pve-2.nh3.internal`, `nh3-pve-2.phasefinal.com` in /etc/hosts). Static
in `/etc/network/interfaces`, plus a UDM reservation on `38:05:25:3b:a0:a8`. `ssh infra-ops@10.100.250.62`
(fleet key, NOPASSWD sudo with `log_output`).
- PVE 9.2.21, installed 2026-10-03 on site to the right disk (the 256 GB Toshiba). **Follow-ups DONE 2026-10-04:** Prime rebooted it at 0724 onto 7.0.14-20; verified 1635: `.62` on vmbr0, `nic0` up at 1 Gb with `arp_ignore=8` + IPv6 off, MeshCentral shows `nh3-pve-2-amt` connected (CIRA, power on), ceph + pve enterprise sources `Enabled: false`, 0 upgradable, no failed units. No guests yet.
- PVE 9.2.21, installed 2026-10-03 on site to the right disk (the 256 GB Toshiba). **Follow-ups DONE 2026-10-04:** Prime rebooted it at 0724 onto 7.0.14-20; verified 1635: `.62` on vmbr0, `nic0` up at 1 Gb with `arp_ignore=8` + IPv6 off, MeshCentral shows `nh3-pve-2-amt` connected (CIRA, power on), ceph + pve enterprise sources `Enabled: false`, 0 upgradable, no failed units.
## Guests
| ID | guest | onboot | notes |
|---|---|---|---|
| 110 | **nh3-albok** (`10.100.50.44`, VLAN 50) | 1 | First guest, 2026-10-04 (albok-dev relaying Prime): runs only albok-service. `servers/nh3-albok/README.md` |
VMIDs continue the NH3 numbering (nh3-pve holds 100–109) so a VMID names one guest on the site.
**Backups:** storage `pbs-ana` (10.250.50.90, datastore `backups`, **namespace `nh3-pve-2`**, token
`root@pam!fleet-vzdump`, secret vaulted as `pbs-ana/fleet-vzdump-token`) and vzdump job **21:30 nightly, `all 1`**, snapshot
mode, fleecing on `vmstore`, the same prune as nh3-pve. 30 min after nh3-pve's 21:00 so the two do not contend on pbs-ana.
pbs-nh3's sync (root namespace, recursive) mirrors the new namespace without any change on the DR side.
⚠ `pvesm add pbs` refuses without `--password` on the command line (it test-logs in), which would put the token in the
sudo log. The stanza was appended to `/etc/pve/storage.cfg` and the secret written to `/etc/pve/priv/storage/pbs-ana.pw`
(37 bytes, value + newline) with `sudo dd` instead.
| NIC | MAC | Chip | Use |
|---|---|---|---|
@@ -17,9 +33,13 @@ possibly a dev environment for security-software work. Standalone node, not clus
⚠ **Keep `auto nic0`.** If Linux leaves the AMT port down, igc powers off the PHY and AMT goes dark. That
happened 2026-10-03 when the bridge moved to the SFP+.
⚠ `vmbr0` is a plain (not VLAN-aware) bridge. Making it VLAN-aware is a separate change: apply it with a reboot,
not a live `ifreload`. On 2026-10-03 a live ifreload that also re-IP'd read the new vmbr0 as "not a bridge" and
downed it (auto-memory `feedback_no_remote_reip_thrash_when_operator_has_console`).
**`vmbr0` is VLAN-aware since 2026-10-04 2311** (`bridge-vlan-aware yes`, `bridge-vids 2-4094`, same as nh3-pve), applied by
`playbooks/nh3-pve-2-vmbr0-vlan-aware.yaml` + a reboot with no guests running. The host address stays untagged (PVID 1 =
the port's native nh3-mgmt); guests take `tag=<vlan>`. Pre-change file kept at `/etc/network/interfaces.pre-vlan-aware-2026-10-04`.
⚠ Change this bridge only by reboot, never a live `ifreload`: on 2026-10-03 a live ifreload that also re-IP'd read the new
vmbr0 as "not a bridge" and downed it (auto-memory `feedback_no_remote_reip_thrash_when_operator_has_console`).
⚠ `ifup -a -s` exits 1 on this host even for a good file: it warns on the PVE-default `bridge-fd 0`. Compare its output
against the previous file's rather than trusting the exit code.
**Disks:** boot = Toshiba 256 GB (`69CA12VQK65N`). **VM storage = LVM-thin `vmstore`, 913 GiB**, on the Crucial 1 TB
(`2610EAD05AB7`). It held the old wrong-drive install, which was wiped 2026-10-03 by `playbooks/nh3-pve-2-disk-prep.yaml`,