diff --git a/configs/restic/README.md b/configs/restic/README.md index caeffe0..99c309c 100644 --- a/configs/restic/README.md +++ b/configs/restic/README.md @@ -33,7 +33,7 @@ On each server, deployed to `/etc/restic/`: - **No secrets in committed config.** `profiles.yaml` references `RESTIC_PASSWORD_FILE=/etc/restic/password` and reads the repository URL through `repository-file: /etc/restic/repository`. Both files live only on the host, root 0400. Both are also vaulted as `/etc/restic/{password,repository}` (2026-09-27). - **Creds go in the URL, not netrc.** restic's rest backend doesn't consult `~/.netrc` — HTTP basic-auth has to be embedded in the repository URL. -- ⚠ **Never `env-file` for the URL.** Until 2026-09-27 the profiles loaded it with `env-file: /etc/restic/restic.env`. But `resticprofile schedule` copies env-file values into the generated systemd units, which are 0644, so every local user could read the rest-server password. All hosts except vm-esh-nas were moved with `playbooks/restic-repository-file.yaml`. vm-esh-nas has no infra-ops account; its script is `configs/restic/vm-esh-nas/migrate-repository-file.sh`, and Prime runs it with sudo. `restic.env` is kept (root 0600, not a leak) because the per-host READMEs and the freshness probe source it. **On a password rotation, update the vault, `restic.env` AND `repository`.** +- ⚠ **Never `env-file` for the URL.** Until 2026-09-27 the profiles loaded it with `env-file: /etc/restic/restic.env`. But `resticprofile schedule` copies env-file values into the generated systemd units, which are 0644, so every local user could read the rest-server password. Every host was moved with `playbooks/restic-repository-file.yaml` (vm-esh-nas once Prime had bootstrapped infra-ops there, the same day). `restic.env` is kept (root 0600, not a leak) because the per-host READMEs and the freshness probe source it. **On a password rotation, update the vault, `restic.env` AND `repository`.** - **Pre-hook runs DB dumps into a staging dir**, then `restic backup` includes that dir alongside the regular paths. One snapshot = one point-in-time. - **`restic forget` is scheduled; `restic prune` is not.** Rest-server's `--append-only` blocks prune from the client side by design. Prune is a manual ceremony (flip the flag, run prune, flip back). diff --git a/configs/restic/vm-esh-nas/migrate-repository-file.sh b/configs/restic/vm-esh-nas/migrate-repository-file.sh deleted file mode 100755 index 29bc806..0000000 --- a/configs/restic/vm-esh-nas/migrate-repository-file.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/bin/bash -# vm-esh-nas: move restic from env-file to repository-file (2026-09-27). -# The same change playbooks/restic-repository-file.yaml made on the seven hosts -# where infra-ops has sudo. vm-esh-nas has no infra-ops account, so run it as: -# ssh -t vm-esh-nas 'sudo bash ~/restic-repofile-migrate.sh' -# It edits the env-file line IN PLACE (the live profile cannot be diffed without -# root, so any drift is preserved), and restores the old profile if a check fails. -set -eu -cd /etc/restic -grep -q '^ *env-file: /etc/restic/restic.env' profiles.yaml \ - || { echo "no env-file line: already migrated, or the profile differs; nothing done"; exit 1; } -val=$(sh -c 'set -a; . /etc/restic/restic.env; printf %s "$RESTIC_REPOSITORY"') -case "$val" in rest:http*) ;; *) echo "RESTIC_REPOSITORY is not a rest: URL; nothing done"; exit 1;; esac -umask 077 -printf '%s\n' "$val" > repository.new -chown root:root repository.new; chmod 0400 repository.new; mv repository.new repository -unset val -cp -p profiles.yaml profiles.yaml.bak-20260927-envfile -trap 'cp -p profiles.yaml.bak-20260927-envfile profiles.yaml; resticprofile --no-ansi --config /etc/restic/profiles.yaml --name default schedule >/dev/null 2>&1 || true; echo "FAILED: old profile restored"' ERR -sed -i 's|^\( *\)env-file: /etc/restic/restic.env.*$|\1repository-file: /etc/restic/repository # not env-file: schedule copies env-file values into world-readable units|' profiles.yaml -resticprofile --no-ansi --config /etc/restic/profiles.yaml --name default cat config >/dev/null -resticprofile --no-ansi --config /etc/restic/profiles.yaml --name default schedule >/dev/null -for u in backup check; do - f=/etc/systemd/system/resticprofile-$u@profile-default.service - test -f "$f" - ! grep -q 'rest:http' "$f" -done -systemctl is-active --quiet resticprofile-backup@profile-default.timer -trap - ERR -echo "vm-esh-nas migrated: units carry no repository URL, backup timer active" diff --git a/docs/runbooks/backups.md b/docs/runbooks/backups.md index eafa9ae..327dabd 100644 --- a/docs/runbooks/backups.md +++ b/docs/runbooks/backups.md @@ -178,7 +178,7 @@ the stop→umount→rm-ghost→remount→start variant. ## Known gaps / TODO -### ✅ `resticprofile schedule` published the repo credential (found and fixed 2026-09-27, except vm-esh-nas) +### ✅ `resticprofile schedule` published the repo credential (found and fixed on all eight hosts, 2026-09-27) On a host whose profile loads `RESTIC_REPOSITORY` from `env-file: /etc/restic/restic.env`, `resticprofile schedule` copies that value, **including @@ -200,11 +200,9 @@ containing `rest:http`. nh3-docker's scheduled unit then ran a real backup (snap `a29b889d`). All seven hosts' URLs and passphrases are now vaulted as `/etc/restic/{repository,password}`. -**Still open: vm-esh-nas.** infra-ops has no account there, so its unit still -leaks. The migration script is staged at `~lkraven/restic-repofile-migrate.sh` -(repo copy `configs/restic/vm-esh-nas/migrate-repository-file.sh`), and Prime runs -it with `ssh -t vm-esh-nas 'sudo bash ~/restic-repofile-migrate.sh'`. Its secrets are -not vaulted yet, because that needs root there. +**vm-esh-nas followed the same day.** Prime bootstrapped infra-ops there +(`playbooks/bootstrap-infra-ops-user.yaml`), and the same playbook then migrated it. +Its secrets are vaulted too. **Zero restic hosts now leak.** The passwords themselves were readable until the move, so the **rotation below is still the real fix**. On rotation, update the vault, `restic.env` and `repository` diff --git a/persistent-memory.md b/persistent-memory.md index dadd33c..cef92e9 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -148,8 +148,8 @@ _As of 2026-09-26 ~1620 PT._ so the systemd units no longer carry the rest-server password. Secrets are vaulted as `/etc/restic/{repository,password}`. `restic.env` is KEPT for manual snippets, so **a rotation must update the vault, `restic.env` and `repository`.** -- **vm-esh-nas is still leaking**, because infra-ops has no account there. Prime runs - `ssh -t vm-esh-nas 'sudo bash ~/restic-repofile-migrate.sh'`. Its secrets are not vaulted. +- **vm-esh-nas done too** (Prime bootstrapped infra-ops there at uid 850, NOPASSWD). **All 8 hosts are clean + and vaulted.** - The passwords were readable until today, so the rotation (Prime's) remains the real fix. ### augaman: face recognition for Cicada diff --git a/servers/vm-esh-nas/README.md b/servers/vm-esh-nas/README.md index 29f4d34..8da0501 100644 --- a/servers/vm-esh-nas/README.md +++ b/servers/vm-esh-nas/README.md @@ -9,7 +9,9 @@ Dozzle agent, and Dockge. - **LAN IP:** 10.0.50.154 - **FQDN:** `vm-esh-nas.esteban.net` -- **SSH:** `lkraven@vm-esh-nas` (key auth; `/etc/hosts` entry on the workstation) +- **SSH:** `infra-ops@10.0.50.154`: the fleet ops identity, uid/gid 850, NOPASSWD sudo, docker group. + Bootstrapped by Prime on 2026-09-27. `lkraven@vm-esh-nas` (the `vm-esh-nas` alias) is the human + account, key auth, with password sudo only. ## Hardware (from latest snapshot) diff --git a/stacks/beszel/README.md b/stacks/beszel/README.md index 879f43e..72d90c3 100644 --- a/stacks/beszel/README.md +++ b/stacks/beszel/README.md @@ -34,8 +34,8 @@ Started by hand, then **fixed the same day**: `hosts/nh3-dev.yaml` no longer bin the NAS shares and `BESZEL_EXTRA_FS` is empty. Their capacity is nh3-nas's own volume, which nh3-nas's agent reports. -Use `infra-ops@` with passwordless sudo, except vm-esh-nas: -`lkraven@10.0.50.154` has Docker access. Irvine's hub address is +Use `infra-ops@` with passwordless sudo. vm-esh-nas has it too since 2026-09-27. +Before that date only `lkraven@10.0.50.154`, with Docker access, worked there. Irvine's hub address is `100.64.0.6`; its retired `10.100.79.3` address caused silent loss of monitoring. ## Filesystems and deployment