fix(restic): stop publishing rest-server passwords in systemd units
resticprofile schedule copies env-file values into the generated units, which
are 0644, so RESTIC_REPOSITORY (the rest-server basic-auth password included)
was readable by every local user on every restic host.
New playbooks/restic-repository-file.yaml:
- derives /etc/restic/repository (root 0400) from restic.env;
- uploads the profile switched to repository-file, but only when the live
profile's sha matches the repo copy it was edited from (drift guard);
- checks the repository is reachable through the new profile (cat config);
- regenerates the units and verifies they exist and contain no rest:http.
Applied to ana-docker, fv-ml1 (configs/restic/ana-ml2), esh-docker-vm,
esh-vm-db, irv-ml1, nh3-dev and nh3-docker. An independent check across all
eight restic hosts (these seven plus esh-ml1) found 0 leaking units. nh3-docker's
scheduled unit ran a real backup afterwards (snapshot a29b889d). Each host's
URL and passphrase are vaulted as <host>/etc/restic/{repository,password}.
restic.env is kept (root 0600) because the per-host READMEs and the freshness
probe source it. A rotation must update the vault, restic.env and repository.
vm-esh-nas has no infra-ops account. Its in-place migration script is staged
for Prime to run with sudo, and its repo profile is pre-edited to match.
Also mirrors augaman-dev's 401df2d (compose header only). The config hash on
esh-ml1 is unchanged.
This commit is contained in:
@@ -22,7 +22,8 @@ On each server, deployed to `/etc/restic/`:
|
||||
├── profiles.yaml # scp'd from configs/restic/<host>/profiles.yaml
|
||||
├── pre-backup.sh # scp'd, 0755, root:root
|
||||
├── password # 0400 root:root — client-side encryption passphrase
|
||||
└── restic.env # 0600 root:root — RESTIC_REPOSITORY=rest:http://user:pw@host:port/path/
|
||||
├── repository # 0400 root:root — rest:http://user:pw@host:port/path/ (read via repository-file)
|
||||
└── restic.env # 0600 root:root — RESTIC_REPOSITORY=… (same URL; kept for manual `. restic.env` snippets)
|
||||
/var/lib/restic/
|
||||
├── stage/ # temp staging for DB dumps; owned by root, 0700
|
||||
└── last-success # unix timestamp of the last successful run
|
||||
@@ -30,8 +31,9 @@ On each server, deployed to `/etc/restic/`:
|
||||
|
||||
## Why this shape
|
||||
|
||||
- **No secrets in committed config.** `profiles.yaml` references `RESTIC_PASSWORD_FILE=/etc/restic/password` and loads `RESTIC_REPOSITORY` from `restic.env`. Both files live only on the host, 0400/0600 root-owned.
|
||||
- **Creds go in the URL, not netrc.** restic's rest backend doesn't consult `~/.netrc` — HTTP basic-auth has to be embedded in the repository URL. Keeping that URL in an env-file (not the committed YAML) means the secret stays on the host.
|
||||
- **No secrets in committed config.** `profiles.yaml` references `RESTIC_PASSWORD_FILE=/etc/restic/password` and reads the repository URL through `repository-file: /etc/restic/repository`. Both files live only on the host, root 0400. Both are also vaulted as `<host>/etc/restic/{password,repository}` (2026-09-27).
|
||||
- **Creds go in the URL, not netrc.** restic's rest backend doesn't consult `~/.netrc` — HTTP basic-auth has to be embedded in the repository URL.
|
||||
- ⚠ **Never `env-file` for the URL.** Until 2026-09-27 the profiles loaded it with `env-file: /etc/restic/restic.env`. But `resticprofile schedule` copies env-file values into the generated systemd units, which are 0644, so every local user could read the rest-server password. All hosts except vm-esh-nas were moved with `playbooks/restic-repository-file.yaml`. vm-esh-nas has no infra-ops account; its script is `configs/restic/vm-esh-nas/migrate-repository-file.sh`, and Prime runs it with sudo. `restic.env` is kept (root 0600, not a leak) because the per-host READMEs and the freshness probe source it. **On a password rotation, update the vault, `restic.env` AND `repository`.**
|
||||
- **Pre-hook runs DB dumps into a staging dir**, then `restic backup` includes that dir alongside the regular paths. One snapshot = one point-in-time.
|
||||
- **`restic forget` is scheduled; `restic prune` is not.** Rest-server's `--append-only` blocks prune from the client side by design. Prune is a manual ceremony (flip the flag, run prune, flip back).
|
||||
|
||||
@@ -66,12 +68,16 @@ ssh -t "$HOST" 'sudo install -d -o root -g root -m 0700 /var/lib/restic/stage'
|
||||
# 2. Seed the two secret files on the host (never in this repo):
|
||||
# - the client-side encryption passphrase (the one used at `restic init`)
|
||||
# - an env-file with the full RESTIC_REPOSITORY URL including HTTP creds
|
||||
# - the same URL alone in /etc/restic/repository (what the profile reads)
|
||||
# Vault them first: secret put <host>/etc/restic/password, …/repository
|
||||
ssh -t "$HOST" 'sudo install -o root -g root -m 0400 /dev/null /etc/restic/password'
|
||||
ssh -t "$HOST" 'sudo install -o root -g root -m 0600 /dev/null /etc/restic/restic.env'
|
||||
ssh -t "$HOST" 'sudo install -o root -g root -m 0400 /dev/null /etc/restic/repository'
|
||||
|
||||
# Seed content (replace <…> with real values from your vault):
|
||||
ssh -t "$HOST" "echo '<encryption-passphrase>' | sudo tee /etc/restic/password >/dev/null"
|
||||
ssh -t "$HOST" "echo 'RESTIC_REPOSITORY=rest:http://<user>:<http-pw>@<rest-server>:8000/<user>/' | sudo tee /etc/restic/restic.env >/dev/null"
|
||||
ssh -t "$HOST" "echo 'rest:http://<user>:<http-pw>@<rest-server>:8000/<user>/' | sudo tee /etc/restic/repository >/dev/null"
|
||||
|
||||
# 3. Push the committed config + pre-hook
|
||||
scp "configs/restic/$HOST/profiles.yaml" "$HOST:/tmp/profiles.yaml"
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
#
|
||||
# Writes to the Anaheim-side rest-server at 10.250.50.70 as user
|
||||
# `ana-docker`. The full REST URL (including HTTP basic-auth creds)
|
||||
# lives in /etc/restic/restic.env — loaded via env-file so this YAML
|
||||
# carries zero secrets and is safe to version-control.
|
||||
# lives in /etc/restic/repository (root 0400), read via repository-file, so
|
||||
# this YAML carries zero secrets and is safe to version-control.
|
||||
#
|
||||
# The client-side encryption passphrase lives in /etc/restic/password.
|
||||
|
||||
@@ -17,7 +17,9 @@ global:
|
||||
min-memory: 100
|
||||
|
||||
default:
|
||||
env-file: /etc/restic/restic.env # provides RESTIC_REPOSITORY=rest:http://user:pw@…
|
||||
repository-file: /etc/restic/repository # URL incl. rest-server creds (root 0400).
|
||||
# NOT env-file: `resticprofile schedule` copies env-file values into the
|
||||
# world-readable systemd units (docs/runbooks/backups.md, Known gaps).
|
||||
env:
|
||||
RESTIC_PASSWORD_FILE: /etc/restic/password
|
||||
initialize: false # repo was created by `restic init`
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
# backup for this host's config/state, so it's the highest-stakes repo in
|
||||
# the fleet. Writes to the Anaheim rest-server at 10.250.50.70 as user
|
||||
# `ana-ml2`. The full REST URL (with HTTP basic auth creds) lives in
|
||||
# /etc/restic/restic.env. The client-side encryption passphrase lives in
|
||||
# /etc/restic/repository (root 0400). The client-side encryption passphrase lives in
|
||||
# /etc/restic/password.
|
||||
#
|
||||
# Notable exclusions:
|
||||
@@ -29,7 +29,9 @@ global:
|
||||
min-memory: 100
|
||||
|
||||
default:
|
||||
env-file: /etc/restic/restic.env # RESTIC_REPOSITORY=rest:http://user:pw@…
|
||||
repository-file: /etc/restic/repository # URL incl. rest-server creds (root 0400).
|
||||
# NOT env-file: `resticprofile schedule` copies env-file values into the
|
||||
# world-readable systemd units (docs/runbooks/backups.md, Known gaps).
|
||||
env:
|
||||
RESTIC_PASSWORD_FILE: /etc/restic/password
|
||||
initialize: false # repo created manually by `restic init`
|
||||
|
||||
@@ -22,7 +22,9 @@ global:
|
||||
min-memory: 100
|
||||
|
||||
default:
|
||||
env-file: /etc/restic/restic.env
|
||||
repository-file: /etc/restic/repository # URL incl. rest-server creds (root 0400).
|
||||
# NOT env-file: `resticprofile schedule` copies env-file values into the
|
||||
# world-readable systemd units (docs/runbooks/backups.md, Known gaps).
|
||||
env:
|
||||
RESTIC_PASSWORD_FILE: /etc/restic/password
|
||||
initialize: false
|
||||
|
||||
@@ -29,13 +29,13 @@ Tested by making it fail (2026-09-27): with a stub `docker` that exits 1 on PATH
|
||||
`resticprofile backup` returned 1, the snapshot count stayed 1 → 1, and
|
||||
`last-success` did not move.
|
||||
|
||||
## ⚠ Deviation from the other hosts: `repository-file`, not `restic.env`
|
||||
## `repository-file`, not `restic.env` (now the fleet standard)
|
||||
|
||||
`resticprofile schedule` copies `env-file` values into the generated systemd
|
||||
units, which are world-readable. On the env-file hosts that publishes the
|
||||
rest-server password to every local user (see `docs/runbooks/backups.md` →
|
||||
Known gaps). Here the URL lives in `/etc/restic/repository` (root 0400) and the
|
||||
unit carries only the path. The playbook's verify step fails if a unit ever
|
||||
units, which are world-readable. That published the rest-server password on
|
||||
every other host until 2026-09-27, when the rest of the fleet was moved to this
|
||||
pattern (see `docs/runbooks/backups.md` → Known gaps). Here the URL lives in
|
||||
`/etc/restic/repository` (root 0400), and the unit carries only the path. The playbook's verify step fails if a unit ever
|
||||
contains `rest:http`.
|
||||
|
||||
## Secrets (seeded by hand; never in this repo)
|
||||
|
||||
@@ -8,13 +8,13 @@
|
||||
# Writes cross-site to rest-server-ana (10.250.50.70:8000/esh-ml1/) because
|
||||
# the ESH site has no local rest-server (same as esh-docker-vm / esh-vm-db).
|
||||
#
|
||||
# ⚠ DELIBERATE DEVIATION from the other hosts: the repository URL (which
|
||||
# embeds the rest-server basic-auth password) is read from
|
||||
# /etc/restic/repository via `repository-file`, NOT from an env-file.
|
||||
# The repository URL (which embeds the rest-server basic-auth password) is
|
||||
# read from /etc/restic/repository via `repository-file`, NOT from an env-file.
|
||||
# `resticprofile schedule` copies env-file values into the generated systemd
|
||||
# unit, and units under /etc/systemd/system are world-readable (0644), so
|
||||
# the env-file pattern publishes the credential to every local user. With
|
||||
# repository-file the unit carries only the path. See README.md.
|
||||
# unit, and units under /etc/systemd/system are world-readable (0644), so the
|
||||
# env-file pattern publishes the credential to every local user. With
|
||||
# repository-file the unit carries only the path. See README.md. (esh-ml1 was
|
||||
# first; the rest of the fleet moved to repository-file on 2026-09-27.)
|
||||
#
|
||||
# ⚠ The CT runs in UTC; the schedules carry an explicit zone so they fire at
|
||||
# the same wall-clock time as the rest of the fleet.
|
||||
|
||||
@@ -30,7 +30,9 @@ global:
|
||||
min-memory: 100
|
||||
|
||||
default:
|
||||
env-file: /etc/restic/restic.env
|
||||
repository-file: /etc/restic/repository # URL incl. rest-server creds (root 0400).
|
||||
# NOT env-file: `resticprofile schedule` copies env-file values into the
|
||||
# world-readable systemd units (docs/runbooks/backups.md, Known gaps).
|
||||
env:
|
||||
RESTIC_PASSWORD_FILE: /etc/restic/password
|
||||
initialize: false
|
||||
|
||||
@@ -33,7 +33,9 @@ global:
|
||||
min-memory: 100
|
||||
|
||||
default:
|
||||
env-file: /etc/restic/restic.env # RESTIC_REPOSITORY=rest:http://user:pw@10.100.50.50:8000/irv-ml1/
|
||||
repository-file: /etc/restic/repository # URL incl. rest-server creds (root 0400).
|
||||
# NOT env-file: `resticprofile schedule` copies env-file values into the
|
||||
# world-readable systemd units (docs/runbooks/backups.md, Known gaps).
|
||||
env:
|
||||
RESTIC_PASSWORD_FILE: /etc/restic/password
|
||||
initialize: false # repo created manually by `restic init`
|
||||
|
||||
@@ -26,7 +26,9 @@ global:
|
||||
min-memory: 200 # workstation has more RAM than agents; give restic a bit more
|
||||
|
||||
default:
|
||||
env-file: /etc/restic/restic.env
|
||||
repository-file: /etc/restic/repository # URL incl. rest-server creds (root 0400).
|
||||
# NOT env-file: `resticprofile schedule` copies env-file values into the
|
||||
# world-readable systemd units (docs/runbooks/backups.md, Known gaps).
|
||||
env:
|
||||
RESTIC_PASSWORD_FILE: /etc/restic/password
|
||||
initialize: false
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
#
|
||||
# Writes to the Synology rest-server at 10.100.50.50:8000 as user
|
||||
# `nh3-docker`. The full REST URL (with HTTP basic-auth creds) lives in
|
||||
# /etc/restic/restic.env. The client-side encryption passphrase lives in
|
||||
# /etc/restic/repository (root 0400). The client-side encryption passphrase lives in
|
||||
# /etc/restic/password.
|
||||
#
|
||||
# No DB dumps needed. None of the stacks on this host (adguard, dockge,
|
||||
@@ -25,7 +25,9 @@ global:
|
||||
min-memory: 100
|
||||
|
||||
default:
|
||||
env-file: /etc/restic/restic.env # RESTIC_REPOSITORY=rest:http://user:pw@…
|
||||
repository-file: /etc/restic/repository # URL incl. rest-server creds (root 0400).
|
||||
# NOT env-file: `resticprofile schedule` copies env-file values into the
|
||||
# world-readable systemd units (docs/runbooks/backups.md, Known gaps).
|
||||
env:
|
||||
RESTIC_PASSWORD_FILE: /etc/restic/password
|
||||
initialize: false # repo created manually by `restic init`
|
||||
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
#!/bin/bash
|
||||
# vm-esh-nas: move restic from env-file to repository-file (2026-09-27).
|
||||
# The same change playbooks/restic-repository-file.yaml made on the seven hosts
|
||||
# where infra-ops has sudo. vm-esh-nas has no infra-ops account, so run it as:
|
||||
# ssh -t vm-esh-nas 'sudo bash ~/restic-repofile-migrate.sh'
|
||||
# It edits the env-file line IN PLACE (the live profile cannot be diffed without
|
||||
# root, so any drift is preserved), and restores the old profile if a check fails.
|
||||
set -eu
|
||||
cd /etc/restic
|
||||
grep -q '^ *env-file: /etc/restic/restic.env' profiles.yaml \
|
||||
|| { echo "no env-file line: already migrated, or the profile differs; nothing done"; exit 1; }
|
||||
val=$(sh -c 'set -a; . /etc/restic/restic.env; printf %s "$RESTIC_REPOSITORY"')
|
||||
case "$val" in rest:http*) ;; *) echo "RESTIC_REPOSITORY is not a rest: URL; nothing done"; exit 1;; esac
|
||||
umask 077
|
||||
printf '%s\n' "$val" > repository.new
|
||||
chown root:root repository.new; chmod 0400 repository.new; mv repository.new repository
|
||||
unset val
|
||||
cp -p profiles.yaml profiles.yaml.bak-20260927-envfile
|
||||
trap 'cp -p profiles.yaml.bak-20260927-envfile profiles.yaml; resticprofile --no-ansi --config /etc/restic/profiles.yaml --name default schedule >/dev/null 2>&1 || true; echo "FAILED: old profile restored"' ERR
|
||||
sed -i 's|^\( *\)env-file: /etc/restic/restic.env.*$|\1repository-file: /etc/restic/repository # not env-file: schedule copies env-file values into world-readable units|' profiles.yaml
|
||||
resticprofile --no-ansi --config /etc/restic/profiles.yaml --name default cat config >/dev/null
|
||||
resticprofile --no-ansi --config /etc/restic/profiles.yaml --name default schedule >/dev/null
|
||||
for u in backup check; do
|
||||
f=/etc/systemd/system/resticprofile-$u@profile-default.service
|
||||
test -f "$f"
|
||||
! grep -q 'rest:http' "$f"
|
||||
done
|
||||
systemctl is-active --quiet resticprofile-backup@profile-default.timer
|
||||
trap - ERR
|
||||
echo "vm-esh-nas migrated: units carry no repository URL, backup timer active"
|
||||
@@ -27,7 +27,7 @@ global:
|
||||
min-memory: 100
|
||||
|
||||
default:
|
||||
env-file: /etc/restic/restic.env
|
||||
repository-file: /etc/restic/repository # not env-file: schedule copies env-file values into world-readable units
|
||||
env:
|
||||
RESTIC_PASSWORD_FILE: /etc/restic/password
|
||||
initialize: false
|
||||
|
||||
Reference in New Issue
Block a user