diff --git a/reference/restic.md b/reference/restic.md new file mode 100644 index 0000000..32aab56 --- /dev/null +++ b/reference/restic.md @@ -0,0 +1,662 @@ +--- +created: '2026-04-16T04:06:58+00:00' +id: 8fc5d55f-1466-4414-860f-ae84c1f39ebf +keywords: +- restic +- backup +- AES-256-CTR +- Poly1305 +- scrypt +- Rabin Fingerprints +- content-defined chunking +- deduplication +- S3 +- B2 +- Azure Blob +- Google Cloud Storage +- rclone +- REST server +- erasure coding +- zstandard +- Blake3 +- snapshot +- prune +- forget +- append-only +links: [] +modified: '2026-04-16T04:06:58+00:00' +summary: Comprehensive reference for restic 0.18.1 — a fast, secure, efficient backup + program written in Go. Covers installation, all storage backends, backup/restore/forget + workflows, encryption, deduplication, repository internals, environment variables, + and troubleshooting. +tags: +- backup +- restic +- encryption +- deduplication +- CLI +- reference +title: Restic Backup Reference +--- + +--- +id: restic +title: Restic Backup Reference +created: 2025-07-10 +modified: 2025-07-10 +tags: + - backup + - restic + - encryption + - deduplication + - CLI + - reference +links: + - reference/docker-compose.md + - reference/crowdsec.md +--- + +# Restic — Backups Done Right + +**Version:** 0.18.1 +**License:** BSD 2-Clause +**Language:** Go +**URL:** https://restic.net/ +**Docs:** https://restic.readthedocs.io/en/stable/ + +## Overview + +Restic is a fast, secure, efficient backup program. It supports Linux, macOS, Windows, FreeBSD, OpenBSD. It is a single binary with no server component required. Key features: + +- **Deduplication** — Content-defined chunking with Rabin Fingerprints; only changed data is transferred/stored +- **Encryption** — AES-256-CTR + Poly1305-AES MAC on all data; scrypt key derivation from user password +- **Compression** — zstandard compression (repository format v2, default since 0.14) +- **Verification** — SHA-256 hashing of all blobs; `restic check` validates integrity +- **Multiple backends** — Local, SFTP, REST, S3, Azure Blob, Google Cloud Storage, Backblaze B2, OpenStack Swift, rclone +- **No server required** — Single static binary; no daemon + +## Installation + +### Package Managers + +| Platform | Command | +|---|---| +| Debian/Ubuntu | `apt-get install restic` | +| Fedora/RHEL/CentOS | `dnf install restic` (or via EPEL for RHEL 8/9) | +| Arch Linux | `pacman -S restic` | +| Alpine | `apk add restic` | +| macOS (Homebrew) | `brew install restic` | +| macOS (MacPorts) | `port install restic` | +| macOS (pkgx) | `pkgx install restic` | +| Nix/NixOS | `nix-env --install restic` | +| FreeBSD | `pkg install restic` | +| OpenBSD | `pkg_add restic` | +| openSUSE | `zypper install restic` | +| Windows (winget) | `winget install --exact --id restic.restic --scope Machine` | +| Windows (Scoop) | `scoop install restic` | +| Solus | `eopkg install restic` | +| Gentoo | `emerge restic` | + +### Official Binaries + +Download from https://github.com/restic/restic/releases — reproducible builds with GPG verification (key `0x91A6868BD3F7A907`). Update in-place with `restic self-update`. + +### Docker + +```bash +docker pull restic/restic +# or from GHCR: +docker pull ghcr.io/restic/restic +``` + +> **Note:** Set a static hostname with `--hostname` when running in Docker. + +### From Source + +Requires Go 1.23+: + +```bash +git clone https://github.com/restic/restic && cd restic && go run build.go +``` + +Produces a statically linked binary. Cross-compilation supported. + +### Autocompletion + +```bash +restic generate --bash-completion /path/to/completion +restic generate --zsh-completion /path/to/completion +restic generate --fish-completion /path/to/completion +restic generate --man /path/to/manpages +restic generate --powershell-completion /path/to/completion +``` + +## Quickstart + +```bash +# Set environment variables +export RESTIC_REPOSITORY=/srv/restic-repo +export RESTIC_PASSWORD=some-strong-password + +# Initialize repository (first time only) +restic init + +# Create backup +restic backup ~/work + +# List snapshots +restic snapshots + +# Restore +restic restore --target /tmp/restore-work your-snapshot-ID + +# Verify integrity +restic check +restic check --read-data # full data verification +``` + +## Preparing a New Repository + +### Repository Versions + +| Version | Min restic | Major Features | +|---|---|---| +| `1` | Any | Original format | +| `2` | 0.14.0+ | Compression support (current default) | + +Specify with `restic init --repository-version 2` or `--repository-version latest`. + +### Password Management Options + +- Environment variable: `RESTIC_PASSWORD` +- Password file: `--password-file` or `RESTIC_PASSWORD_FILE` +- Password command: `--password-command` or `RESTIC_PASSWORD_COMMAND` +- Interactive prompt (default) + +### Backend Types + +#### Local + +```bash +restic init --repo /srv/restic-repo +``` + +#### SFTP + +```bash +restic -r sftp:user@host:/srv/restic-repo init +# With port and IPv6: +restic -r sftp://user@[::1]:2222//srv/restic-repo init +# Use SSH config host aliases: +restic -r sftp:my-backup-host:/srv/restic-repo init +# Custom SFTP command: +restic -r sftp:foo:/repo -o sftp.command="foobar" +``` + +SSH keepalive recommended: + +``` +ServerAliveInterval 60 +ServerAliveCountMax 240 +``` + +#### REST Server + +```bash +restic -r rest:http://host:8000/ init +restic -r rest:https://user:pass@host:8000/ init +restic -r rest:http+unix:///tmp/rest.socket:/repo init +``` + +TLS cert override: `--cacert`. Env vars: `RESTIC_REST_USERNAME`, `RESTIC_REST_PASSWORD`. + +#### Amazon S3 + +```bash +export AWS_ACCESS_KEY_ID= +export AWS_SECRET_ACCESS_KEY= +export AWS_SESSION_TOKEN= # for temporary credentials +restic -r s3:s3.us-east-1.amazonaws.com/bucket_name init +``` + +> **Note:** restic uses **path-style URLs**, not virtual-hosted style. + +#### S3-Compatible (Minio, Wasabi, Alibaba OSS, etc.) + +```bash +export AWS_ACCESS_KEY_ID= +export AWS_SECRET_ACCESS_KEY= +restic -r s3:https://server:port/bucket_name init +# Options: +# -o s3.region="us-east-1" +# -o s3.bucket-lookup=auto|dns|path +# -o s3.list-objects-v1=true # for old Ceph +``` + +#### Backblaze B2 + +```bash +# Recommended: use S3-compatible API (see B2 docs for S3 keys) +# Legacy B2 backend (less reliable error handling): +export B2_ACCOUNT_ID= +export B2_ACCOUNT_KEY= +restic -r b2:bucketname:path/to/repo init +``` + +#### Microsoft Azure Blob Storage + +```bash +export AZURE_ACCOUNT_NAME= +export AZURE_ACCOUNT_KEY= # or AZURE_ACCOUNT_SAS= +# Or use: az login (needs Storage Blob Data Contributor role) +restic -r azure:container:/ init +# Options: -o azure.connections=10, -o azure.access-tier=Hot|Cool|Cold +``` + +#### Google Cloud Storage + +```bash +export GOOGLE_PROJECT_ID= +export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json +restic -r gs:bucket:/ init +# Options: -o gs.connections=10, -o gs.region=us +``` + +#### OpenStack Swift + +```bash +# Keystone v1/v2/v3 auth via environment variables +# (ST_AUTH, ST_USER, ST_KEY for v1; OS_AUTH_URL, OS_USERNAME, etc. for v2/v3) +restic -r swift:container_name:/path init +``` + +#### rclone (any supported remote) + +```bash +restic -r rclone:remote_name:path init +# Options: -o rclone.program, -o rclone.args, -o rclone.timeout +# Remote rclone via SSH: +restic -o rclone.program="ssh user@remotehost rclone" -r rclone:b2:foo/bar +``` + +### Group-Accessible Repositories + +Since v0.14, local and SFTP repos can be shared with a group: + +```bash +# Read-only: chmod -R g+r /srv/restic-repo +# Read-write: requires root to set setgid +# find /srv/restic-repo -type d -exec chmod g+s '{}' \; +# chmod -R g+rw /srv/restic-repo +``` + +## Backing Up + +### Basic Backup + +```bash +restic -r /repo backup ~/work +restic -r /repo backup ~/work.txt # individual files +``` + +### File Change Detection + +On Unix, a file is considered unchanged if **all** of these match a previous backup: + +- Modification timestamp (mtime) +- Metadata change timestamp (ctime) +- File size +- Inode number + +Options to modify: + +- `--force` — rescan all files +- `--ignore-ctime` — only require mtime match +- `--ignore-inode` — allow inode and ctime to differ (for FUSE filesystems) + +Parent snapshot selection defaults to `--group-by host,paths`. + +### Skip Unchanged Snapshots + +```bash +restic backup ~/work --skip-if-unchanged +``` + +### Dry Run + +```bash +restic backup ~/work --dry-run -vv +``` + +### Excluding Files + +Options: + +- `--exclude "pattern"` / `--iexclude "pattern"` (case-insensitive) +- `--exclude-file file` / `--iexclude-file file` +- `--exclude-caches` — skip folders with CACHEDIR.TAG +- `--exclude-if-present foo` — skip folders containing file `foo` +- `--exclude-larger-than 1M` — skip files over size +- `--exclude-cloud-files` — skip cloud-only files (Windows) + +Pattern syntax uses Go's `filepath.Match`: + +- `foo` matches `/dir/foo` and `/dir/foo/file` but not `/dir/foobar` +- Leading `/` anchors to root +- `**` matches across directories +- `!pattern` cancels a previous match (like gitignore) +- Environment variables expanded with `$VAR` +- `--one-file-system` prevents crossing mount boundaries + +### Including Files + +```bash +restic backup --files-from /path/to/patterns +restic backup --files-from-verbatim /path/to/list # no glob expansion +restic backup --files-from-raw /path/to/list # NUL-delimited +``` + +### Comparing Snapshots + +```bash +restic diff snapshot1 snapshot2 +restic diff snapshot1:/subfolder snapshot2:/subfolder +restic diff snapshot1 snapshot2 --metadata +``` + +Status codes: `+` added, `-` removed, `T` type changed, `M` content changed, `U` metadata changed, `?` bitrot detected. + +### Backing Up from Commands/Stdin + +```bash +# Recommended: from command (detects failures) +restic backup --stdin-filename db.sql --stdin-from-command -- mysqldump --host example mydb + +# Stdin pipe (masks command failures — use with set -o pipefail) +mysqldump mydb | restic backup --stdin --stdin-filename db.sql +``` + +### Tags + +```bash +restic backup --tag projectX --tag foo ~/work +``` + +Tags used for snapshot selection with `forget` and `tag` commands. + +### Windows VSS + +```bash +restic backup C:\ --use-fs-snapshot +# Options: +# -o vss.timeout=2m30s +# -o vss.exclude-all-mount-points=true +# -o vss.exclude-volumes="d:;c:\mnt" +``` + +### Exit Status Codes + +- `0` — success (snapshot created with all files) +- `1` — fatal error (no snapshot created) +- `3` — some source files unreadable (incomplete snapshot created) + +### Scheduling Backups + +Restic has no built-in scheduler. Options: + +- **Linux/BSD:** systemd timers, cron +- **Windows:** Task Scheduler +- **Wrapper tools:** resticprofile + +Always detect already-running instances before starting a backup. + +## Environment Variables + +| Variable | Purpose | +|---|---| +| `RESTIC_REPOSITORY` | Repository location | +| `RESTIC_REPOSITORY_FILE` | File containing repository location | +| `RESTIC_PASSWORD` | Repository password | +| `RESTIC_PASSWORD_FILE` | File containing password | +| `RESTIC_PASSWORD_COMMAND` | Command to print password | +| `RESTIC_CACHE_DIR` | Cache directory location | +| `RESTIC_COMPRESSION` | Compression mode (v2 only) | +| `RESTIC_HOST` | Set/override hostname for snapshots | +| `RESTIC_PACK_SIZE` | Target pack file size | +| `RESTIC_READ_CONCURRENCY` | File read concurrency | +| `RESTIC_PROGRESS_FPS` | Progress bar refresh rate | +| `RESTIC_CACERT` | TLS CA certificate(s) | +| `RESTIC_TLS_CLIENT_CERT` | TLS client certificate | +| `RESTIC_KEY_HINT` | Key ID to try first | +| `AWS_ACCESS_KEY_ID` | S3 access key | +| `AWS_SECRET_ACCESS_KEY` | S3 secret key | +| `AWS_SESSION_TOKEN` | S3 temporary session token | +| `AWS_DEFAULT_REGION` | S3 default region | +| `AWS_PROFILE` | AWS credentials profile | +| `AZURE_ACCOUNT_NAME` | Azure account name | +| `AZURE_ACCOUNT_KEY` | Azure account key | +| `AZURE_ACCOUNT_SAS` | Azure SAS token | +| `GOOGLE_PROJECT_ID` | GCP project ID | +| `GOOGLE_APPLICATION_CREDENTIALS` | GCP service account JSON key | +| `B2_ACCOUNT_ID` | Backblaze B2 account ID | +| `B2_ACCOUNT_KEY` | Backblaze B2 application key | +| `RESTIC_REST_USERNAME` | REST server username | +| `RESTIC_REST_PASSWORD` | REST server password | + +## Working with Repositories + +### Listing Snapshots + +```bash +restic snapshots +restic snapshots --group-by host,paths # default +restic snapshots --host myhost --tag foo +``` + +### Listing Files in a Snapshot + +```bash +restic ls latest +restic ls snapshot-id /path/within/snapshot +``` + +### Restoring + +```bash +restic restore --target /tmp/restore snapshot-id +restic restore --target /tmp/restore --include "*.txt" snapshot-id +restic restore --target /tmp/restore --exclude "*.c" snapshot-id +# Mount as filesystem: +restic mount /mnt/restic +# Print to stdout: +restic dump snapshot-id path/to/file +``` + +### Copying Snapshots Between Repositories + +```bash +restic copy --repo-copy /other-repo snapshot-id +``` + +### Removing Files from Snapshots + +```bash +restic forget --keep-last 5 snapshot-id # remove from repo +restic rewrite --exclude "pattern" snapshot-id # create new snapshot without matching files +``` + +### Modifying Snapshot Metadata + +```bash +restic tag --tag newtag snapshot-id +restic tag --remove tagname snapshot-id +``` + +### Checking Integrity + +```bash +restic check # check metadata +restic check --read-data # verify all data +restic check --check-unused # find unused blobs +``` + +## Removing Backup Snapshots + +### Manual Removal + +```bash +restic forget snapshot-id +restic prune # cleanup unreferenced data +``` + +### Policy-Based Removal + +```bash +restic forget --keep-last 7 --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-yearly 5 --prune +``` + +Policy options: + +| Option | Description | +|---|---| +| `--keep-last N` | Keep last N snapshots | +| `--keep-hourly N` | Keep most recent snapshot for each of last N hours with snapshots | +| `--keep-daily N` | Keep most recent snapshot for each of last N days with snapshots | +| `--keep-weekly N` | Keep most recent snapshot for each of last N weeks | +| `--keep-monthly N` | Keep most recent snapshot for each of last N months | +| `--keep-yearly N` | Keep most recent snapshot for each of last N years | +| `--keep-tag TAG` | Keep all snapshots with specified tag(s) | +| `--keep-within DURATION` | Keep snapshots within duration of latest (e.g., `2y5m7d3h`) | +| `--keep-within-{hourly,daily,weekly,monthly,yearly} DURATION` | Duration-based variants | + +Options work on natural time boundaries (weeks are Mon–Sun, hours are :00–:59). Results are ORed — a snapshot only needs to match one option to be kept. Grouped by `--group-by host,paths` by default. + +All policies support `--dry-run`. `--unsafe-allow-remove-all` allows removing all snapshots for a specific filter (v0.17+). + +### Append-Only Mode Security + +When using append-only repositories (e.g., via rest-server), use `--keep-within` policies. Calendar-based policies (`--keep-daily`, etc.) can be gamed by attackers injecting fake snapshots. `--keep-within` keeps both legitimate and attacker snapshots, allowing detection. + +### Customizing Pruning + +```bash +restic prune # default: 5% max unused +restic prune --max-unused 200M # absolute size limit +restic prune --max-unused 10% # relative to repo size +restic prune --max-unused unlimited # never repack (fast, but wastes space) +restic prune --max-repack-size 10G # limit total repack size +restic prune --repack-cacheable-only # only repack cached metadata (fast) +restic prune --repack-small # repack packs below 80% target size +restic prune --repack-smaller-than 5M # repack packs below this size +``` + +### Recovering from "No Free Space" + +```bash +# Step 1: forget some snapshots +restic forget --keep-last 1 +# Step 2: minimal prune +restic prune --max-repack-size 0 +# Step 3: if still stuck (last resort): +restic prune --unsafe-recover-no-free-space +``` + +## Tuning Backup Parameters + +| Parameter | Option | Default | +|---|---|---| +| Progress estimation | `--no-scan` / `RESTIC_PROGRESS_FPS` | Enabled | +| Backend connections | `-o backend.connections=N` | 5 | +| CPU usage | `GOMAXPROCS=N` env var | All cores | +| Compression | `--compression auto\|max\|off` | `auto` | +| Data verification | `--insecure-no-verify` (dangerous) | Enabled | +| File read concurrency | `--read-concurrency N` or `RESTIC_READ_CONCURRENCY` | 2 | +| Pack size | `--pack-size N` or `RESTIC_PACK_SIZE` | 8 MiB | +| Feature flags | `-o feature.=true` | — | + +## Encryption + +- **Algorithm:** AES-256-CTR + Poly1305-AES MAC +- **KDF:** scrypt (N=65536, r=8, p=1) +- **Key management:** Multiple keys supported; password change doesn't require re-encryption +- All data encrypted except informational metadata in key files +- Cache is also encrypted + +### Key Management + +```bash +restic key list +restic key add +restic key remove +restic key passwd +restic key set # set default key hint +``` + +## Repository Design + +### Repository Layout + +``` +restic-repo/ +├── config # encrypted JSON (version, ID, chunker params) +├── data/ # pack files (two-char prefix dirs) +│ ├── 21/ +│ ├── 32/ +│ └── ... +├── index/ # index files mapping blobs to packs +├── keys/ # encrypted key files (scrypt-derived) +├── locks/ # lock files for concurrent access +├── snapshots/ # snapshot metadata +└── tmp/ # temporary files +``` + +### Pack Format + +Each pack file contains: + +``` +EncryptedBlob1 || ... || EncryptedBlobN || EncryptedHeader || Header_Length +``` + +Blob types: `0b00` data, `0b01` tree, `0b10` compressed data, `0b11` compressed tree (v2 only). + +### Deduplication + +Content-defined chunking with Rabin Fingerprints (64-byte sliding window). Blob sizes: 512 KiB to 8 MiB (target 1 MiB average). Only modified blobs stored in subsequent backups — works even with byte insertions/removals at arbitrary positions. + +### Snapshots + +Each backup creates a snapshot — a JSON document recording paths, hostname, username, time, tags, and the root tree hash. Snapshots are immutable once created. + +### Locking + +Two lock types: exclusive (for prune/rewrite) and non-exclusive (for backup/restore). Locks older than 30 minutes are considered stale. Use `--retry-lock ` to wait for locks. + +## Scripting + +```bash +# JSON output +restic snapshots --json +restic backup ~/work --json + +# Check if repo is initialized +restic cat config + +# Exit codes for scripting +# 0 = success, 1 = fatal error, 3 = partial (some files unreadable) +# 10 = repo locked, 11 = connection error (and others) +``` + +## Troubleshooting + +1. Find what's damaged: `restic check` +2. Backup the repository (copy data files) +3. Repair index: `restic repair index` +4. Run all backups +5. Remove missing data from snapshots: `restic repair snapshots` +6. Check again: `restic check` + +## See Also + +- [Docker Compose Reference](reference/docker-compose.md) +- [CrowdSec Reference](reference/crowdsec.md) \ No newline at end of file