Homelab Topology Evolution: A Half-Year Retrospective

From my first Linux lab host in February 2025 to July 2026, this homelab has grown from "SSH in and install some stuff" into a small, layered setup. I'm writing this half-year architecture review (the real span is closer to a year and a half) mostly for future me: why I wired things this way, which decisions held up, and what I plan to dismantle next. If you're stacking machines too, hold it up against your own blast-radius checklist.

What the Layering Looks Like Now

Roughly six layers, top to bottom. Each layer gets a clear job, instead of one old host playing router, storage, games, and photo library until the day it all goes down together.

Network layer: modem bridged, OpenWrt as the gateway, VLANs separating guests, IoT, and trusted clients, AdGuard Home handling home DNS and basic filtering.
Access layer: day-to-day remote access runs over WireGuard or Headscale; Cloudflare Tunnel gets enabled only when an HTTP service needs a small, controlled slice of the public internet — not as a default tunnel-everything setup.
Compute layer: Proxmox slicing LXC containers and VMs; a separate Docker host runs the app stacks; a small VPS in the cloud stays as a jump host and partial reverse proxy, so port forwarding on home broadband is never the only way in.
Storage layer: ZFS mirror on TrueNAS, datasets split by purpose — photos, backup target, media, lab volumes — each with its own snapshot policy instead of one big pool with sloppy bookkeeping.
Observability layer: Prometheus plus Grafana, disk SMART health checks, logs from critical hosts and the reverse proxy; alerts go out over an external channel, not the wobbly links inside the lab.
Backup layer: ZFS snapshots plus restic or borg offsite; game saves and config repos follow their own cadence, and restore drills have been spot-tested at least.

The topology lives in git as a draw.io diagram plus a plain-text sketch; every big change gets a new version, and the commit message says why. Three months later, when you're debugging "why can this box reach that box," the diagram beats memory every time. A topology with no diagram eventually becomes folklore passed by word of mouth.

What Went Right This Half-Year

Backups before fancy services. Offsite backups of photos and config outrank opening another pretty dashboard, always. After the one near-panic over a disk warning, this became a hard rule: before any new service goes live, ask where the data lands, whether it can be rebuilt if lost, and whether the backup actually covers it.

Network isolation came early. IoT and guests never share the trusted segment, and camera plus NAS management ports are unreachable by default. Adding Frigate, a game server, or a friend's guest access later was just opening a controlled hole in an existing boundary — not tearing everything down to draw one bigger flat L2 network.

Config in Git. Compose files, key OpenWrt snippets, Headscale ACL drafts, alert rules — there's no perfect CI, but "can diff, can roll back, can rebuild on another machine" has already saved multiple fumbles and disk mishaps.

Split the blast radius. Game servers sit apart from core data; lab containers carry a delete-by date; the management plane stays tucked behind tunnels instead of a public port hiding behind an "unusual" number. The control plane itself — the Headscale database, for instance — gets production-grade backups, not "it's a small service, it'll be fine."

The Detours I Definitely Took

Early on, one all-purpose Docker box meant a single impulsive docker compose down could take out a whole string of services, and debugging couldn't tell whether the app or the network policy had broken. Too many plugins and containers, then maintenance fatigue, then zombie containers squatting on disk and ports that nobody dared delete. For a while backups were local-only; when the SMART alert fired I discovered the offsite link had never actually worked — the job reported success, but the target was another disk in the same rack.

Running too many experiment tracks at once was just as bad: k3s to try, a new panel, a self-hosted email itch, another monitoring stack — with no retirement criteria for any of them. The rack and the DNS end up full of half-finished things, and the cognitive load hurts more than CPU usage. Writing your detours into a retrospective isn't shameful; it's where the next phase's delete list comes from. No delete list, and the lab only ever grows one way.

Next Phase (H2 2026 Intentions)

Keys and rotation: audit service accounts, tunnel credentials, and backup passwords, and set a rotation cadence — instead of "it's in the password manager, so it's permanently safe."
DR numbers: write down executable RTO and RPO for photos, the config repo, and critical VMs, and run one restore drill for each. Borg and restic are a start — now it needs to land on the calendar.
Subtraction: list every service with no access in thirty days and shut it down or delete it by default; every dashed node on the topology is either drawn solid as something someone maintains, or erased.
Docs: keep filling in out-of-band management steps, ACL rollbacks, disk replacement, and certificate renewal procedures; keep diagrams in sync with the repo, so we never end up with one outdated diagram living only in my head.

If I Could Keep Only Three Things

Offsite backups, network isolation, config in Git. Everything else can be rebuilt — dashboards reinstalled, game servers restarted. A pretty rack is not the goal; services being rebuildable and data being unkillable is what makes stacking more machines worth it.

Looking back at that early-2025 host where everything piled up, the biggest win isn't having more machines — it's that layering shrank the failure domains. A game server blowing up doesn't take the photo library with it, and a compromised guest network can't reach the storage management port directly. If next phase only gets one thing done, I'd rather turn DR numbers and restore drills into a repeatable habit than introduce yet another orchestration system.