BorgBackup: Incremental Backups and a Restore Drill

restic is the backup tool I know better, but in June I deliberately took Borg through a "we actually need to restore" run — because one machine at home has always used a Borg repository, and its strategy can't fight restic's. You can have two tools; your authoritative datasets and runbooks just can't contradict each other. What follows is the after-action report: drill goals, timeline, where it broke, and which docs got changed. Only a restore you can actually perform is a backup — I'd written that sentence a dozen times, and it only came alive again when I got stuck on a passphrase environment variable.

Drill goals (written down in advance)

• List archives from the last seven days in the remote SSH repository
• Restore only /etc/stack and one app-data directory to a temporary path
• Verify with checksums, or by actually opening the config files
• Record wall-clock time and every blocker
• Don't touch live mounts or production directories

I deliberately didn't attempt a full-disaster recovery on the first pass: prove that the single-directory path works before talking bare metal. Writing the goals down means the drill doesn't drift into "let me tidy up the repo" or "let me upgrade the client while I'm here" — widen the scope and you'll never finish.

State beforehand

Repo initialization (a historical command, not re-run on drill day):

borg init --encryption=repokey user@backuphost:~/borg-lab

Routine creation:

borg create --stats \
  user@backuphost:~/borg-lab::'{hostname}-{now}' \
  /data /etc/stack /opt/stack

The password lives in my password manager. repokey mode means the key material is stored encrypted on the repo side — lose the passphrase and the archive is worthless. Before the drill I confirmed a second copy of the passphrase exists, which should honestly have been step zero of the checklist. If you use keyfile mode, also make sure the key file isn't stuck in the dead loop of "only existing on the backed-up host."

The timeline

Minute zero, list the archives. borg list succeeded on the first try, the newest archives are named correctly, mood is good.

Minute five, extract. The plan:

mkdir -p /tmp/borg-drill && cd /tmp/borg-drill
borg extract user@backuphost:~/borg-lab::ARCHIVE-ID etc/stack

Where it actually stalled: SSH multiplexing and the agent are fine in non-interactive cron environments, but I'd switched users inside tmux — permissions weren't right and BORG_PASSPHRASE wasn't exported. It failed twice before I pointed BORG_PASSCOMMAND at a readable script. The docs only covered create, never the environment variables for extract — that's documentation debt number one, and the most likely thing to freeze you in a real incident.

Minute twenty, verification. The restored docker-compose.yml matched the live hash. But one cert path was a symlink, and it came out of extract dangling — if a service pointed straight at it, it would break. I added a checklist item: after a restore, check that symlink targets are inside the backup set. Some paths actually depend on other directories on the machine, so a single-directory restore is inherently incomplete; the runbook needs the dependency boundaries written down.

Minute thirty-five, FUSE mount comparison. borg mount is much lighter than extract for browsing, and it's the right tool when you need "just one file." I forgot fusermount -u when unmounting, and the temp mount point lingered until the next day — carelessness number two, and proof that a drill has to include cleanup steps, not just "found the file, we win."

Minute fifty, cleanup. Deleted the temp directory, confirmed no plaintext keys landed on disk, confirmed the mount point was gone.

All told, about an hour of effective work including the mishaps. Under real incident pressure, hunting for the passphrase and permissions alone would've added another half hour — so the passphrase and restore steps have to be practiced outside of incidents, written into a checklist someone else can follow too.

What broke, and the fixes

1. The runbook had backups but no restores. Fixed: list → extract or mount → verify → clean up, with environment variables and common failure modes included.
2. Symlinks and sparse files. Critical paths now either back up the actual content or have their external dependencies documented.
3. Repo host security. Least privilege for the backup server account, password login disabled, source restrictions in place — a compromised repo means your history is stolen or deleted.
4. The Borg/restic division of labor is now in the checklist. Photos and general directories go through restic; the Borg repo stays Borg; and no dataset gets written by both tools without an explicit note, so prune policies can't sabotage each other's intent.

Compression, encryption, and retention

Default compression is usually good enough — make sure you can restore first, tune the level later. Pick the encryption mode you can actually walk through a restore path with, not the trendiest term you've heard. Align the prune policy with restic's keep rules in "calendar language": for example, keep seven daily, four weekly, six monthly, so you're not deleting aggressively on one side while assuming the offsite is full. On drill day I also ran borg check once for repo integrity — better to find a problem then than on disaster night.

Wrap-up

borg create's green stats prove nothing; opening a critical file from a temp directory fifty minutes later and being able to state the elapsed time and the gotchas is what means this week's backup is still valid. The next drill will use a different repo and restore only a single database dump, hunting for the next documentation gap. A backup system's maturity isn't measured by how many successful create logs you've written — it's whether you dare to put the next drill date on the calendar.

If you also run more than one backup tool: write down "which data, which tool, which repo, who has the keys" before arguing about which one is more advanced. Strategy conflicts that cause accidental deletes and prunes are far scarier than any performance difference between the tools. The biggest win from this Borg drill wasn't learning some subcommand — it was turning the restore steps from "I know how" into "it's written down."