Create: reference/restic.md

2026-04-15 21:06:58 -07:00
parent c204f1d934
commit d994526207
1 changed files with 662 additions and 0 deletions
@@ -0,0 +1,662 @@
+---
+created: '2026-04-16T04:06:58+00:00'
+id: 8fc5d55f-1466-4414-860f-ae84c1f39ebf
+keywords:
+- restic
+- backup
+- AES-256-CTR
+- Poly1305
+- scrypt
+- Rabin Fingerprints
+- content-defined chunking
+- deduplication
+- S3
+- B2
+- Azure Blob
+- Google Cloud Storage
+- rclone
+- REST server
+- erasure coding
+- zstandard
+- Blake3
+- snapshot
+- prune
+- forget
+- append-only
+links: []
+modified: '2026-04-16T04:06:58+00:00'
+summary: Comprehensive reference for restic 0.18.1 — a fast, secure, efficient backup
+  program written in Go. Covers installation, all storage backends, backup/restore/forget
+  workflows, encryption, deduplication, repository internals, environment variables,
+  and troubleshooting.
+tags:
+- backup
+- restic
+- encryption
+- deduplication
+- CLI
+- reference
+title: Restic Backup Reference
+---
+
+---
+id: restic
+title: Restic Backup Reference
+created: 2025-07-10
+modified: 2025-07-10
+tags:
+  - backup
+  - restic
+  - encryption
+  - deduplication
+  - CLI
+  - reference
+links:
+  - reference/docker-compose.md
+  - reference/crowdsec.md
+---
+
+# Restic — Backups Done Right
+
+**Version:** 0.18.1
+**License:** BSD 2-Clause
+**Language:** Go
+**URL:** https://restic.net/
+**Docs:** https://restic.readthedocs.io/en/stable/
+
+## Overview
+
+Restic is a fast, secure, efficient backup program. It supports Linux, macOS, Windows, FreeBSD, OpenBSD. It is a single binary with no server component required. Key features:
+
+- **Deduplication** — Content-defined chunking with Rabin Fingerprints; only changed data is transferred/stored
+- **Encryption** — AES-256-CTR + Poly1305-AES MAC on all data; scrypt key derivation from user password
+- **Compression** — zstandard compression (repository format v2, default since 0.14)
+- **Verification** — SHA-256 hashing of all blobs; `restic check` validates integrity
+- **Multiple backends** — Local, SFTP, REST, S3, Azure Blob, Google Cloud Storage, Backblaze B2, OpenStack Swift, rclone
+- **No server required** — Single static binary; no daemon
+
+## Installation
+
+### Package Managers
+
+| Platform | Command |
+|---|---|
+| Debian/Ubuntu | `apt-get install restic` |
+| Fedora/RHEL/CentOS | `dnf install restic` (or via EPEL for RHEL 8/9) |
+| Arch Linux | `pacman -S restic` |
+| Alpine | `apk add restic` |
+| macOS (Homebrew) | `brew install restic` |
+| macOS (MacPorts) | `port install restic` |
+| macOS (pkgx) | `pkgx install restic` |
+| Nix/NixOS | `nix-env --install restic` |
+| FreeBSD | `pkg install restic` |
+| OpenBSD | `pkg_add restic` |
+| openSUSE | `zypper install restic` |
+| Windows (winget) | `winget install --exact --id restic.restic --scope Machine` |
+| Windows (Scoop) | `scoop install restic` |
+| Solus | `eopkg install restic` |
+| Gentoo | `emerge restic` |
+
+### Official Binaries
+
+Download from https://github.com/restic/restic/releases — reproducible builds with GPG verification (key `0x91A6868BD3F7A907`). Update in-place with `restic self-update`.
+
+### Docker
+
+```bash
+docker pull restic/restic
+# or from GHCR:
+docker pull ghcr.io/restic/restic
+```
+
+> **Note:** Set a static hostname with `--hostname` when running in Docker.
+
+### From Source
+
+Requires Go 1.23+:
+
+```bash
+git clone https://github.com/restic/restic && cd restic && go run build.go
+```
+
+Produces a statically linked binary. Cross-compilation supported.
+
+### Autocompletion
+
+```bash
+restic generate --bash-completion /path/to/completion
+restic generate --zsh-completion /path/to/completion
+restic generate --fish-completion /path/to/completion
+restic generate --man /path/to/manpages
+restic generate --powershell-completion /path/to/completion
+```
+
+## Quickstart
+
+```bash
+# Set environment variables
+export RESTIC_REPOSITORY=/srv/restic-repo
+export RESTIC_PASSWORD=some-strong-password
+
+# Initialize repository (first time only)
+restic init
+
+# Create backup
+restic backup ~/work
+
+# List snapshots
+restic snapshots
+
+# Restore
+restic restore --target /tmp/restore-work your-snapshot-ID
+
+# Verify integrity
+restic check
+restic check --read-data  # full data verification
+```
+
+## Preparing a New Repository
+
+### Repository Versions
+
+| Version | Min restic | Major Features |
+|---|---|---|
+| `1` | Any | Original format |
+| `2` | 0.14.0+ | Compression support (current default) |
+
+Specify with `restic init --repository-version 2` or `--repository-version latest`.
+
+### Password Management Options
+
+- Environment variable: `RESTIC_PASSWORD`
+- Password file: `--password-file` or `RESTIC_PASSWORD_FILE`
+- Password command: `--password-command` or `RESTIC_PASSWORD_COMMAND`
+- Interactive prompt (default)
+
+### Backend Types
+
+#### Local
+
+```bash
+restic init --repo /srv/restic-repo
+```
+
+#### SFTP
+
+```bash
+restic -r sftp:user@host:/srv/restic-repo init
+# With port and IPv6:
+restic -r sftp://user@[::1]:2222//srv/restic-repo init
+# Use SSH config host aliases:
+restic -r sftp:my-backup-host:/srv/restic-repo init
+# Custom SFTP command:
+restic -r sftp:foo:/repo -o sftp.command="foobar"
+```
+
+SSH keepalive recommended:
+
+```
+ServerAliveInterval 60
+ServerAliveCountMax 240
+```
+
+#### REST Server
+
+```bash
+restic -r rest:http://host:8000/ init
+restic -r rest:https://user:pass@host:8000/ init
+restic -r rest:http+unix:///tmp/rest.socket:/repo init
+```
+
+TLS cert override: `--cacert`. Env vars: `RESTIC_REST_USERNAME`, `RESTIC_REST_PASSWORD`.
+
+#### Amazon S3
+
+```bash
+export AWS_ACCESS_KEY_ID=<KEY>
+export AWS_SECRET_ACCESS_KEY=<SECRET>
+export AWS_SESSION_TOKEN=<TOKEN>  # for temporary credentials
+restic -r s3:s3.us-east-1.amazonaws.com/bucket_name init
+```
+
+> **Note:** restic uses **path-style URLs**, not virtual-hosted style.
+
+#### S3-Compatible (Minio, Wasabi, Alibaba OSS, etc.)
+
+```bash
+export AWS_ACCESS_KEY_ID=<KEY>
+export AWS_SECRET_ACCESS_KEY=<SECRET>
+restic -r s3:https://server:port/bucket_name init
+# Options:
+# -o s3.region="us-east-1"
+# -o s3.bucket-lookup=auto|dns|path
+# -o s3.list-objects-v1=true  # for old Ceph
+```
+
+#### Backblaze B2
+
+```bash
+# Recommended: use S3-compatible API (see B2 docs for S3 keys)
+# Legacy B2 backend (less reliable error handling):
+export B2_ACCOUNT_ID=<ID>
+export B2_ACCOUNT_KEY=<KEY>
+restic -r b2:bucketname:path/to/repo init
+```
+
+#### Microsoft Azure Blob Storage
+
+```bash
+export AZURE_ACCOUNT_NAME=<NAME>
+export AZURE_ACCOUNT_KEY=<KEY>     # or AZURE_ACCOUNT_SAS=<SAS>
+# Or use: az login (needs Storage Blob Data Contributor role)
+restic -r azure:container:/ init
+# Options: -o azure.connections=10, -o azure.access-tier=Hot|Cool|Cold
+```
+
+#### Google Cloud Storage
+
+```bash
+export GOOGLE_PROJECT_ID=<ID>
+export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json
+restic -r gs:bucket:/ init
+# Options: -o gs.connections=10, -o gs.region=us
+```
+
+#### OpenStack Swift
+
+```bash
+# Keystone v1/v2/v3 auth via environment variables
+# (ST_AUTH, ST_USER, ST_KEY for v1; OS_AUTH_URL, OS_USERNAME, etc. for v2/v3)
+restic -r swift:container_name:/path init
+```
+
+#### rclone (any supported remote)
+
+```bash
+restic -r rclone:remote_name:path init
+# Options: -o rclone.program, -o rclone.args, -o rclone.timeout
+# Remote rclone via SSH:
+restic -o rclone.program="ssh user@remotehost rclone" -r rclone:b2:foo/bar
+```
+
+### Group-Accessible Repositories
+
+Since v0.14, local and SFTP repos can be shared with a group:
+
+```bash
+# Read-only: chmod -R g+r /srv/restic-repo
+# Read-write: requires root to set setgid
+# find /srv/restic-repo -type d -exec chmod g+s '{}' \;
+# chmod -R g+rw /srv/restic-repo
+```
+
+## Backing Up
+
+### Basic Backup
+
+```bash
+restic -r /repo backup ~/work
+restic -r /repo backup ~/work.txt  # individual files
+```
+
+### File Change Detection
+
+On Unix, a file is considered unchanged if **all** of these match a previous backup:
+
+- Modification timestamp (mtime)
+- Metadata change timestamp (ctime)
+- File size
+- Inode number
+
+Options to modify:
+
+- `--force` — rescan all files
+- `--ignore-ctime` — only require mtime match
+- `--ignore-inode` — allow inode and ctime to differ (for FUSE filesystems)
+
+Parent snapshot selection defaults to `--group-by host,paths`.
+
+### Skip Unchanged Snapshots
+
+```bash
+restic backup ~/work --skip-if-unchanged
+```
+
+### Dry Run
+
+```bash
+restic backup ~/work --dry-run -vv
+```
+
+### Excluding Files
+
+Options:
+
+- `--exclude "pattern"` / `--iexclude "pattern"` (case-insensitive)
+- `--exclude-file file` / `--iexclude-file file`
+- `--exclude-caches` — skip folders with CACHEDIR.TAG
+- `--exclude-if-present foo` — skip folders containing file `foo`
+- `--exclude-larger-than 1M` — skip files over size
+- `--exclude-cloud-files` — skip cloud-only files (Windows)
+
+Pattern syntax uses Go's `filepath.Match`:
+
+- `foo` matches `/dir/foo` and `/dir/foo/file` but not `/dir/foobar`
+- Leading `/` anchors to root
+- `**` matches across directories
+- `!pattern` cancels a previous match (like gitignore)
+- Environment variables expanded with `$VAR`
+- `--one-file-system` prevents crossing mount boundaries
+
+### Including Files
+
+```bash
+restic backup --files-from /path/to/patterns
+restic backup --files-from-verbatim /path/to/list  # no glob expansion
+restic backup --files-from-raw /path/to/list  # NUL-delimited
+```
+
+### Comparing Snapshots
+
+```bash
+restic diff snapshot1 snapshot2
+restic diff snapshot1:/subfolder snapshot2:/subfolder
+restic diff snapshot1 snapshot2 --metadata
+```
+
+Status codes: `+` added, `-` removed, `T` type changed, `M` content changed, `U` metadata changed, `?` bitrot detected.
+
+### Backing Up from Commands/Stdin
+
+```bash
+# Recommended: from command (detects failures)
+restic backup --stdin-filename db.sql --stdin-from-command -- mysqldump --host example mydb
+
+# Stdin pipe (masks command failures — use with set -o pipefail)
+mysqldump mydb | restic backup --stdin --stdin-filename db.sql
+```
+
+### Tags
+
+```bash
+restic backup --tag projectX --tag foo ~/work
+```
+
+Tags used for snapshot selection with `forget` and `tag` commands.
+
+### Windows VSS
+
+```bash
+restic backup C:\ --use-fs-snapshot
+# Options:
+# -o vss.timeout=2m30s
+# -o vss.exclude-all-mount-points=true
+# -o vss.exclude-volumes="d:;c:\mnt"
+```
+
+### Exit Status Codes
+
+- `0` — success (snapshot created with all files)
+- `1` — fatal error (no snapshot created)
+- `3` — some source files unreadable (incomplete snapshot created)
+
+### Scheduling Backups
+
+Restic has no built-in scheduler. Options:
+
+- **Linux/BSD:** systemd timers, cron
+- **Windows:** Task Scheduler
+- **Wrapper tools:** resticprofile
+
+Always detect already-running instances before starting a backup.
+
+## Environment Variables
+
+| Variable | Purpose |
+|---|---|
+| `RESTIC_REPOSITORY` | Repository location |
+| `RESTIC_REPOSITORY_FILE` | File containing repository location |
+| `RESTIC_PASSWORD` | Repository password |
+| `RESTIC_PASSWORD_FILE` | File containing password |
+| `RESTIC_PASSWORD_COMMAND` | Command to print password |
+| `RESTIC_CACHE_DIR` | Cache directory location |
+| `RESTIC_COMPRESSION` | Compression mode (v2 only) |
+| `RESTIC_HOST` | Set/override hostname for snapshots |
+| `RESTIC_PACK_SIZE` | Target pack file size |
+| `RESTIC_READ_CONCURRENCY` | File read concurrency |
+| `RESTIC_PROGRESS_FPS` | Progress bar refresh rate |
+| `RESTIC_CACERT` | TLS CA certificate(s) |
+| `RESTIC_TLS_CLIENT_CERT` | TLS client certificate |
+| `RESTIC_KEY_HINT` | Key ID to try first |
+| `AWS_ACCESS_KEY_ID` | S3 access key |
+| `AWS_SECRET_ACCESS_KEY` | S3 secret key |
+| `AWS_SESSION_TOKEN` | S3 temporary session token |
+| `AWS_DEFAULT_REGION` | S3 default region |
+| `AWS_PROFILE` | AWS credentials profile |
+| `AZURE_ACCOUNT_NAME` | Azure account name |
+| `AZURE_ACCOUNT_KEY` | Azure account key |
+| `AZURE_ACCOUNT_SAS` | Azure SAS token |
+| `GOOGLE_PROJECT_ID` | GCP project ID |
+| `GOOGLE_APPLICATION_CREDENTIALS` | GCP service account JSON key |
+| `B2_ACCOUNT_ID` | Backblaze B2 account ID |
+| `B2_ACCOUNT_KEY` | Backblaze B2 application key |
+| `RESTIC_REST_USERNAME` | REST server username |
+| `RESTIC_REST_PASSWORD` | REST server password |
+
+## Working with Repositories
+
+### Listing Snapshots
+
+```bash
+restic snapshots
+restic snapshots --group-by host,paths  # default
+restic snapshots --host myhost --tag foo
+```
+
+### Listing Files in a Snapshot
+
+```bash
+restic ls latest
+restic ls snapshot-id /path/within/snapshot
+```
+
+### Restoring
+
+```bash
+restic restore --target /tmp/restore snapshot-id
+restic restore --target /tmp/restore --include "*.txt" snapshot-id
+restic restore --target /tmp/restore --exclude "*.c" snapshot-id
+# Mount as filesystem:
+restic mount /mnt/restic
+# Print to stdout:
+restic dump snapshot-id path/to/file
+```
+
+### Copying Snapshots Between Repositories
+
+```bash
+restic copy --repo-copy /other-repo snapshot-id
+```
+
+### Removing Files from Snapshots
+
+```bash
+restic forget --keep-last 5 snapshot-id  # remove from repo
+restic rewrite --exclude "pattern" snapshot-id  # create new snapshot without matching files
+```
+
+### Modifying Snapshot Metadata
+
+```bash
+restic tag --tag newtag snapshot-id
+restic tag --remove tagname snapshot-id
+```
+
+### Checking Integrity
+
+```bash
+restic check                    # check metadata
+restic check --read-data         # verify all data
+restic check --check-unused      # find unused blobs
+```
+
+## Removing Backup Snapshots
+
+### Manual Removal
+
+```bash
+restic forget snapshot-id
+restic prune  # cleanup unreferenced data
+```
+
+### Policy-Based Removal
+
+```bash
+restic forget --keep-last 7 --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-yearly 5 --prune
+```
+
+Policy options:
+
+| Option | Description |
+|---|---|
+| `--keep-last N` | Keep last N snapshots |
+| `--keep-hourly N` | Keep most recent snapshot for each of last N hours with snapshots |
+| `--keep-daily N` | Keep most recent snapshot for each of last N days with snapshots |
+| `--keep-weekly N` | Keep most recent snapshot for each of last N weeks |
+| `--keep-monthly N` | Keep most recent snapshot for each of last N months |
+| `--keep-yearly N` | Keep most recent snapshot for each of last N years |
+| `--keep-tag TAG` | Keep all snapshots with specified tag(s) |
+| `--keep-within DURATION` | Keep snapshots within duration of latest (e.g., `2y5m7d3h`) |
+| `--keep-within-{hourly,daily,weekly,monthly,yearly} DURATION` | Duration-based variants |
+
+Options work on natural time boundaries (weeks are Mon–Sun, hours are :00–:59). Results are ORed — a snapshot only needs to match one option to be kept. Grouped by `--group-by host,paths` by default.
+
+All policies support `--dry-run`. `--unsafe-allow-remove-all` allows removing all snapshots for a specific filter (v0.17+).
+
+### Append-Only Mode Security
+
+When using append-only repositories (e.g., via rest-server), use `--keep-within` policies. Calendar-based policies (`--keep-daily`, etc.) can be gamed by attackers injecting fake snapshots. `--keep-within` keeps both legitimate and attacker snapshots, allowing detection.
+
+### Customizing Pruning
+
+```bash
+restic prune                           # default: 5% max unused
+restic prune --max-unused 200M         # absolute size limit
+restic prune --max-unused 10%          # relative to repo size
+restic prune --max-unused unlimited    # never repack (fast, but wastes space)
+restic prune --max-repack-size 10G     # limit total repack size
+restic prune --repack-cacheable-only   # only repack cached metadata (fast)
+restic prune --repack-small            # repack packs below 80% target size
+restic prune --repack-smaller-than 5M  # repack packs below this size
+```
+
+### Recovering from "No Free Space"
+
+```bash
+# Step 1: forget some snapshots
+restic forget --keep-last 1
+# Step 2: minimal prune
+restic prune --max-repack-size 0
+# Step 3: if still stuck (last resort):
+restic prune --unsafe-recover-no-free-space <ID>
+```
+
+## Tuning Backup Parameters
+
+| Parameter | Option | Default |
+|---|---|---|
+| Progress estimation | `--no-scan` / `RESTIC_PROGRESS_FPS` | Enabled |
+| Backend connections | `-o backend.connections=N` | 5 |
+| CPU usage | `GOMAXPROCS=N` env var | All cores |
+| Compression | `--compression auto\|max\|off` | `auto` |
+| Data verification | `--insecure-no-verify` (dangerous) | Enabled |
+| File read concurrency | `--read-concurrency N` or `RESTIC_READ_CONCURRENCY` | 2 |
+| Pack size | `--pack-size N` or `RESTIC_PACK_SIZE` | 8 MiB |
+| Feature flags | `-o feature.<name>=true` | — |
+
+## Encryption
+
+- **Algorithm:** AES-256-CTR + Poly1305-AES MAC
+- **KDF:** scrypt (N=65536, r=8, p=1)
+- **Key management:** Multiple keys supported; password change doesn't require re-encryption
+- All data encrypted except informational metadata in key files
+- Cache is also encrypted
+
+### Key Management
+
+```bash
+restic key list
+restic key add
+restic key remove <ID>
+restic key passwd
+restic key set <ID>  # set default key hint
+```
+
+## Repository Design
+
+### Repository Layout
+
+```
+restic-repo/
+├── config          # encrypted JSON (version, ID, chunker params)
+├── data/           # pack files (two-char prefix dirs)
+│   ├── 21/
+│   ├── 32/
+│   └── ...
+├── index/          # index files mapping blobs to packs
+├── keys/           # encrypted key files (scrypt-derived)
+├── locks/          # lock files for concurrent access
+├── snapshots/      # snapshot metadata
+└── tmp/            # temporary files
+```
+
+### Pack Format
+
+Each pack file contains:
+
+```
+EncryptedBlob1 || ... || EncryptedBlobN || EncryptedHeader || Header_Length
+```
+
+Blob types: `0b00` data, `0b01` tree, `0b10` compressed data, `0b11` compressed tree (v2 only).
+
+### Deduplication
+
+Content-defined chunking with Rabin Fingerprints (64-byte sliding window). Blob sizes: 512 KiB to 8 MiB (target 1 MiB average). Only modified blobs stored in subsequent backups — works even with byte insertions/removals at arbitrary positions.
+
+### Snapshots
+
+Each backup creates a snapshot — a JSON document recording paths, hostname, username, time, tags, and the root tree hash. Snapshots are immutable once created.
+
+### Locking
+
+Two lock types: exclusive (for prune/rewrite) and non-exclusive (for backup/restore). Locks older than 30 minutes are considered stale. Use `--retry-lock <duration>` to wait for locks.
+
+## Scripting
+
+```bash
+# JSON output
+restic snapshots --json
+restic backup ~/work --json
+
+# Check if repo is initialized
+restic cat config
+
+# Exit codes for scripting
+# 0 = success, 1 = fatal error, 3 = partial (some files unreadable)
+# 10 = repo locked, 11 = connection error (and others)
+```
+
+## Troubleshooting
+
+1. Find what's damaged: `restic check`
+2. Backup the repository (copy data files)
+3. Repair index: `restic repair index`
+4. Run all backups
+5. Remove missing data from snapshots: `restic repair snapshots`
+6. Check again: `restic check`
+
+## See Also
+
+- [Docker Compose Reference](reference/docker-compose.md)
+- [CrowdSec Reference](reference/crowdsec.md)