restic, Done the 3-2-1 Way: Local and Offsite Backups

Everyone nods along when backups come up; it's when something actually breaks that you discover the script never ran, the password's been forgotten, or you've never restored once. The tool fight (restic vs. Borg vs. whatever) is mostly noise. What actually decides whether you get home is whether the strategy is complete, whether the keys are retrievable, and whether you've drilled the restore on a quiet day. I've seen a repository full of tidy snapshots where the owner couldn't find the password paper — that wasn't a tool failure, it was a philosophy missing the "a human can still open this" step. I treat 3-2-1 as the constraint and restic as the implementation. Constraints first, then the commands.

3-2-1 isn't a slogan; it's three testable claims

3 copies: live data, local backup, and offsite — at least one of each, so losing one isn't fatal
2 media types: e.g. a NAS disk and cloud object storage, so they don't share the same failure domain
1 offsite copy: the one fire, theft, or a whole-rack power/water event can't reach

Mapped to my home: live data on the working disk; a restic local repository on a USB/spare disk or secondary NAS; an offsite repository on Backblaze B2 or any S3-compatible store. restic does the encryption and dedup so two repositories don't double the disk usage. The key file gets permissions 600 and a printed or offline copy sealed away — if the key and the repository vanish together, an encrypted backup is just very pretty random noise.

"Testable" means you can point at directories and a bucket and say where the three copies live, not a vague "it's in the cloud too."

Trade-offs before the philosophy hits the ground: what to back up, what to skip on purpose

Back up the things whose loss would make you cry and whose rebuild cost exceeds their storage cost: compose and configs like /opt/stack, database logical-dump directories, important libraries, and the .ssh plus notes in your home. Deliberately skip: caches, container image layers, re-downloadable ISOs, and worktrees that can be re-cloned from git with no unpushed local changes. Be honest when you put "can re-download" in the exclude list: in an offline environment, or if the upstream disappears, an ISO stops being a re-acquirable asset.

If your media library is measured in terabytes, give it its own policy and cadence rather than sharing one forget parameter set with daily-changing configs. For databases, prefer "dump first, then restic" to avoid consistency debates over copying the data directory directly. Under the same philosophy, password vault exports, certificate private keys, and tunnel credentials belong in the encrypted backup — but with stricter permissions and placement than ordinary documents. "restic is encrypted anyway" is not a license to scatter them in world-readable directories.

Initialize two repositories

export RESTIC_PASSWORD_FILE=/root/.restic-pass
chmod 600 /root/.restic-pass

export RESTIC_REPOSITORY=/mnt/backup/restic-lab
restic init

export RESTIC_REPOSITORY=s3:https://s3.example.com/bucket/lab
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
restic init

The local and remote can be two repositories, or you can back up locally and restic copy to the remote. I prefer running backup to both sides on the critical paths: the structure stays symmetric and troubleshooting skips the "did the copy finish?" state machine. Environment variables live in a root-readable env file loaded by systemd via EnvironmentFile=, not pasted into a world-readable cron line.

Scheduling, tags, and retention: keep the timeline readable

restic backup /opt/stack /var/backups/db \
  --tag daily --exclude-file=/etc/restic/excludes.txt

restic forget --prune \
  --keep-daily 7 \
  --keep-weekly 4 \
  --keep-monthly 6

forget without --prune only drops references; actual space reclaim needs prune. Whether production scripts run them in one pass depends on the lock window you can accept. Tags make it easy to filter snapshots per workload. Failures must notify — silent failure is the cancer of backups; a non-zero exit code belongs in an email or your phone channel.

Run backups under a dedicated system user with read-only access to the data: the minimum politeness of least privilege. If your cloud supports object lock, turn it on so a compromised key being abused by ransomware can't delete your backups. Cost-wise, B2 and friends are usually fine for a home budget; the expensive part is egress when restoring — think through where a big restore happens.

No drill, no backup

Set aside half a day each quarter:

restic snapshots
restic restore latest --target /tmp/restore-test --include /opt/stack
# spot-check hashes or try docker compose up in an empty dir

Record how long it took, which keys were missing, and whether DNS and firewall blocked S3. Proving backup exited 0 isn't enough; you need to prove you can come back from zero. The usual drill findings: the password file lives only in an encrypted home dir; an offsite key rotation left an old timer untouched; or excludes are so broad the library never actually got backed up.

One more operating discipline people skip: after changing exclude rules, rotating passwords, or moving repository paths, run a short backup plus restic check to confirm the repo index is still healthy. If the offsite repository has been getting forget but no new backup for a long time, suspect the timer points at the wrong RESTIC_REPOSITORY. When multiple machines share one repository, plan hostname and path tags in advance, or your snapshots list turns into an archaeology site.