Blog

Prometheus + Grafana: Keeping an Eye on the Machines at Home

May 27, 2025 5 min read
MonitoringOpsHomelab
Prometheus and Grafana monitoring dashboards in a homelab

Before monitoring went live, my approach was "ssh in and run top after something breaks." It broke twice: a disk filled to 100% and took the database down with it, and the CPU ran hot enough to throttle without me having a clue. For me, Prometheus + Grafana was never a KPI wall — it's about getting woken up less often by surprises. So this isn't an install manual; it's an observation log — what I actually click on every day, and why I still keep it around.

A lot of people install a row of exporters and call it done. I care more about the version of me three weeks out: will I still open this thing, and what's the first thing I look at? If the answer is "no idea what to look at" or "all red dots, whatever," then the monitoring might as well not exist. An observation log forces me to record actual usage honestly, instead of showing off an architecture diagram.

Observation log: where the components sit in the rack

For a home setup, a single Prometheus instance is plenty. Keep its data directory on an SSD and retain 15–30 days. Don't fight the media library for random I/O on the spinning disk.

Fixed roles:

  • Prometheus: scraping and storage
  • Grafana: the entry point I actually "look at" every day
  • node_exporter: every Linux box
  • Optional: cAdvisor, smartctl exporter, Blackbox

Scrape snippet:

scrape_configs:
  - job_name: node
    static_configs:
      - targets:
          - lab-core:9100
          - nas:9100
          - vps:9100

On the VPS, the exporter listens only on the WireGuard address, never the public interface. Security groups and firewalls allow by subnet. Once I mistakenly opened 9100 to 0.0.0.0 and got a pile of node metrics scanned from the public internet — free information disclosure. Pulled it back into the WG subnet the same day.

Every morning: the four panels I check before the coffee cools

The order barely changes and takes under three minutes.

First: disk free space. Root partition and /data. I flag >85% yellow and >92% red. A full disk is the accident I've actually lived through — logs, Docker layers, and backup staging can quietly eat everything. On the panel I show big "GB remaining" numbers, not pretty curves: curves are for post-mortems, big numbers are for decisions.

Second: memory and swap. Spikes are fine; sustained swapping is not. Backup jobs hammering the cache overnight is understandable; if business hours show chronic swap, it's time to drop a container or add RAM. Swap fooled me once: load looked fine, but Minecraft stuttered on every autosave — disk and memory were suffering together.

Third: CPU iowait and load. Learn to tell "busy computing" from "dragged down by disk." Compiles, ZFS scrubs, and full-disk rsyncs raise load as expected. If iowait is high and disk utilization is pegged, go find who's writing — it's usually a backup that forgot its rate limit.

Fourth: are the critical services up? node up, key compose stacks, game server processes, or Blackbox probing a port. A process can be alive while its port isn't listening; watching host liveness alone gives you a fake green. I once added a Blackbox probe for lab-core:25565 so I'd catch "systemd active but Minecraft stuck on startup."

After those four, if everything's green, the phone goes down. If something's yellow, only then do I drill into finer graphs: which mount, which disk, which job stopped scraping. The daily discipline is "don't expand by default" — otherwise Grafana turns into a new time sink.

Random afternoons: when I do take a second look

Most weekdays I don't reopen Grafana after lunch. Exceptions:

  • Disk was yellow in the morning; check whether it kept climbing
  • Someone at home complains "the internet feels slow" — look at NAS NIC throughput and whether the router is even in my monitoring scope (often it isn't; that's a separate debt)
  • I just changed a compose stack; check uptime and error rate after restart

The dashboard started as the official node dashboard, then I deleted half the panels I never used. A fancy percentile curve doesn't beat a big "GB left on disk." I once added a dozen "cool" graphs and found two weeks later that I only ever looked at four — everything else got collapsed and deleted, and loading got faster.

On my phone I keep exactly one dashboard link, named "three minutes after waking up." The name is corny, but it means future me never gets lost in a folder of dashboards.

Alert log: what deserves to interrupt me

Alertmanager pushes to Telegram. The rules are minimal:

  • Disk space
  • Instance down for more than 5 minutes
  • SMART predictive failure (if present)
  • Certificate expiry (if probeable)

I don't alert on "CPU > 80% for 1 minute." Compiles and backups would trip it constantly, and once people learn to ignore alerts, they ignore the important ones too. The moment noise goes up, notifications get muted. First rule of home monitoring: if it rings, it has to be worth standing up for.

For one week I also alerted on "container restarted," and my phone blew up the night Docker updated. Deleted the rule the next day. A restart by itself isn't an incident; a restart storm is — that needs a steadier signal, not a push on every up -d.

Night-shift log: how the monitoring ate itself

Retention was once set to 180 days, and Prometheus's TSDB on a small SSD wrote the disk up to the alert line. The monitoring component monitoring itself into the red — Ironic. Fix: cut retention to 30 days, compress/clean up, and add Prometheus's data directory as an explicit mount under "disk first glance."

A scrape interval of 15–30s is plenty. High-cardinality labels (usernames, full URLs) will blow up memory — stay modest in a home setup. Raspberry-Pi-class boxes especially need restraint; don't drag the company's cardinality budget back home.

Grafana anonymous read-only is fine, but don't expose it raw to the internet; keep it behind WireGuard or wrap it in Cloudflare Access. I briefly exposed Grafana publicly "for phone convenience," and the login attempts in the logs had me pulling it back the same night.

Deployment and omissions: notes at the end of the log

Things I've tripped on, and now check during startup inspection:

  • Is the node_exporter firewall open to the scraping source?
  • Are timezones consistent, so alerts don't look like they're from the future?
  • Am I only monitoring hosts and not business ports?
  • Does the backup job overlap with a scrub in the same window and peg the disk?

Monitoring is the ops nervous system. Cover disk, downtime, and critical ports first; worry about pretty dashboards later. If it buys you an hour between "something's wrong" and "disaster," it's already paid for its electricity. What I actually look at each day is very little — but those few glances have to be real.

That's today's observation log. No new dashboards, no new alert rules. Sometimes the healthiest state of monitoring is going two weeks straight thinking "there's nothing interesting to look at" — that means the foundation is still online.

← Back to Blog