Configuring Lab Linux Machines at Scale with Ansible

Once you cross three machines, hand-editing config over SSH starts to rot: this box got fail2ban, that one didn't; timezones are right on some and wrong on others. In March I folded my common provisioning into an Ansible playbook. This post isn't "installing Ansible from zero" — it's about how the inventory grew into what it is today. The evolution history is more useful than the final files.

Prehistory: one hosts file and a prayer

The earliest inventory was three lines of IPs, the username passed on the command line with -u, and every host named 10.8.0.x. The day the ping module finally ran I was thrilled; the next day I'd forgotten which machine was the NAS and which was the experiment box. Lesson one: an inventory without readable names is an inventory without documentation.

[lab]
10.8.0.10
10.8.0.20
10.8.0.5

Key-based login has to be sorted out first. Batch password login is the opening scene of a disaster movie — when a key dies, stop and fix it, don't write passwords into shell history. Once ansible_host lined up with the WireGuard LAN addresses, I got into the habit of running batches from my laptop over the tunnel, instead of throwing playbooks directly at public IPs.

Phase one: groups and ansible_host

Splitting "what it's called" from "where I connect" instantly lowered the mental load:

[lab]
lab-core ansible_host=10.8.0.10
nas-app  ansible_host=10.8.0.20
vps-1    ansible_host=10.8.0.5

[lab:vars]
ansible_user=veritas
ansible_python_interpreter=/usr/bin/python3

[lab:vars] killed the repetitive -u and interpreter path. Then came the itch for [lab:children]: split out nas, vps, and workstation, because the security baseline is the same but the docker and backup roles aren't. The driver of every evolution is always the same: a --limit typed at the wrong scope, or a role that shouldn't be running on that box at all.

I also tried splitting host variables into host_vars/: timezone uniform, hostname per-host, docker enabled as a boolean. The inventory itself got thinner, the differences moved into the file tree, and code review made "what's actually special about this box" visible. The downside is that someone cloning the repo has one more directory convention to learn, so the README spells out "which files to edit for a new machine."

Phase two: roles grow in, the inventory starts growing labels

The everyday roles stay boring:

common: timezone, hostname, base packages, vim
security: ssh hardening, fail2ban, ufw skeleton
docker: the engine and the current user's group
node_exporter: the monitoring agent

Boring is what makes a role repeatable across machines. On the inventory side I started adding label-style groups — [docker_hosts], [monitored] — and a host can appear in several groups. That reads far better than a long when: inventory_hostname in ... chain in the playbook. Sensitive variables go into ansible-vault, and the vault password is never the same as the login password.

Phase three: check, limit, tags become muscle memory

ansible-playbook -i inventory site.yml --check
ansible-playbook -i inventory site.yml --limit lab-core
ansible-playbook -i inventory site.yml --tags security
ansible-playbook -i inventory site.yml

Check first, then apply. Production changes get a --limit; never carpet-bomb the whole fleet at once. Plays get tags so I can run just the security bits. Handlers that restart sshd deserve caution — make sure they won't kick out every session. I once locked myself into the gap between "new config not yet active" and "old sessions already dead" on an experiment box. These days it's an explicit confirmation window.

The inventory file itself goes into Git, but production keys and vault ciphertext have to be handled cleanly; plaintext secrets in the repo are the dark ages of this evolution, and they only had to happen once.

The boundary: write it into a role on the third time

One-off experiments stay manual. Anything that repeats a third time becomes a role. If you fold a temporary experiment into a force-applied role, three months later you'll get "converged" without warning — the port silently closed by ufw again, the test user deleted. That's why the inventory has carried [lab_unmanaged] or skip-groups: declaring "this box isn't being converged right now" is more reliable than remembering it.

The payoff of automation: a new machine is in monitoring, in backups, and on the same security baseline within half an hour. That consistency matters more than any cool task. The onboarding flow got fixed too: set up SSH keys and sudo first, add the box to the right inventory groups, run --limit with common and security, and only add docker or the exporter once that's clean. And one comment line per host — "purpose / owner / safe to force-converge" — is a gift to the you of three months from now.

The inventory grew from three IPs into a small map with groups, variables, and an exception list. It's still growing, but at least I no longer have to remember who 10.8.0.20 is. The evolution continues — maybe a multi-environment inventory next, or some facts generated by a dynamic script. The principles don't change: readable names, limitable scope, declarable exceptions, and no plaintext secrets in the repo.

One real "half-failed" batch

One time I wanted to install qemu-guest-agent and fix the timezone on five machines at once. The playbook was tiny, --check passed, and the real run got three successes and two failures: one box had the wrong Python interpreter path (a slim container-style LXC), and one had SSH switched to key-only while I'd temporarily been using a password parameter.

The fix wasn't "re-scan the world with -vvv." It was:

Use --limit to re-run only the failed hosts
Give the LXC group its own ansible_python_interpreter
Feed the password parameter in from a command-line environment variable, not the repo

Batch tools amplify small oversights. The real value of Ansible in a homelab is usually freezing the steps that work, not orchestrating a hundred machines in one go.

Keys and sensitive variables

Passwords in group_vars moved to Ansible Vault or out of git entirely, with a vars.local.yml left on the machine (mode 600). I once committed an SMTP app password in plaintext in a role, pushed it to my private Gitea, and told myself "it's private, fine." I rotated it anyway. Private does not mean leak-free.

Restraint applies to handlers that restart services: if a reload is enough for an sshd config change, don't restart. Stacked firewall rule errors plus a restart can boot you out of your own machine — when you're editing network-related plays, keep an active session open. Same discipline as sshd.

Blog