Disk Health Monitoring: smartctl and Early-Warning Scripts

Disks don't pick a convenient time to die. After I wired SMART patrol checks across every data disk in the lab last May, the alert finally fired before the disaster instead of after. This post isn't an attribute encyclopedia; it's a few real (and nearly real) failing-disk and misjudgment stories that explain what to watch, how to report it, and when to pull the trigger on a replacement. When you're done, you should be able to write an executable threshold for your own cron — not go re-google the parameter table.

Case A: pending sectors go from 0 to "can't gamble"

A 4TB SATA drive three years in, at a weekly check:

sudo smartctl -a /dev/sde | egrep 'Reallocated|Pending|Offline_Uncorrectable|Temperature'

Current_Pending_Sector went from 0 last week to 8, while Reallocated_Sector_Ct stayed at 0. No I/O errors reported, ZFS wasn't complaining either. A lot of people would "keep an eye on it" — we didn't: same day we confirmed the mirror's data redundancy was healthy, ordered a same-capacity drive, and replaced it via the pool procedure. After it came out, a long test showed pending sectors still climbing. If we'd waited for Offline_Uncorrectable while the other side coughed at the same time, the story would've been a lot uglier.

Lesson: pending sectors that stay non-zero and keep climbing are a replace-the-drive signal, not decorative alerting. SMART's whole value is giving you a window before the filesystem starts screaming.

Case B: occasional temperature vs. reallocation trend

One drive in a summer rack corner hit 48°C raw, and the alert pushed to Telegram. Looking at load: daytime scrub plus game-server save backups stacked together. After adding simple airflow and moving the scrub off peak hours, steady state settled to 38–40°C. For the next two weeks the reallocation counter stayed at 0.

Contrast with Case A: temperature spikes are usually reversible; monotonically climbing reallocations and pending counts mostly aren't. Alert on both, but prioritize differently — temperature first means checking airflow, chassis layout, and task peaks; sector-class metrics go straight into a replacement decision, no "let's give it another week" negotiations.

Case C: the USB enclosure that "looks healthy"

A cold spare in a USB 3 enclosure sometimes returned only partial SMART attributes, or was only visible briefly after power-on. Once when the enclosure's power was flaky, the system saw I/O errors while SMART read back near-normal "old snapshot" values — almost misdiagnosed as a cable problem. After moving the critical data disks to direct motherboard SATA or HBA passthrough, the attributes and self-test logs became trustworthy.

Lesson: USB enclosures don't always pass SMART through intact; don't build a critical pool behind an "enclosure filter." Virtual disks on hardware RAID cards hide member-drive truth the same way — one of the reasons I prefer HBA passthrough plus ZFS. If a cold spare can only live in USB, at least accept that "SMART is for reference," and validate the backup by actually restoring, not by a PASSED label.

Case D: SSD life percentage lies, and also saves you

A SATA SSD as the system disk: when metrics like Percentage Used hit 85%, random writes started stuttering while SMART still reported PASSED overall. We replaced it early after assessing capacity and write model, and demoted the old drive to throwaway experiments. PASSED only means "not yet failed by the vendor's definition," not "you can safely write for another year." Consumer-grade life estimates tend to run optimistic, and if your lab writes a lot of logs and container layers, watch the trend, not the pass line.

Separately, an NVMe showed media-error counter jumps after a power loss — visible in both vendor tools and smartctl -x. With no redundancy on a system disk, I imaged the whole drive and swapped it the same day, refusing to "wait since it still boots."

Daily commands and an automation skeleton

sudo smartctl -a /dev/sdX
sudo smartctl -t short /dev/sdX
sudo smartctl -l selftest /dev/sdX
# long test in the maintenance window: sudo smartctl -t long /dev/sdX

Watch trends, not single magic numbers (vendor attribute numbers differ):

Reallocated_Sector_Ct, Current_Pending_Sector, Offline_Uncorrectable
Sustained high temperature, not a one-off spike
SSD remaining life and total writes
Error logs and self-tests showing FAILED

Use smartd or a cron script; on anomaly push to Telegram or email (email via external SMTP, please — don't bind it to your own wobbly mail stack). ZFS users, run scrub in parallel — SMART sees the disk, scrub sees the pool; one reports media, the other reports checksums, and they complement each other. Keep a simple history of patrol results (even a CSV) or you won't see the slope from 0 to 8.

Replacement rules (the kind you put in your ops notes)

Pending or reallocated counts keep climbing → prepare to replace; don't gamble
With redundancy: confirm the other side is healthy first, then follow the pool's replace procedure
Without redundancy: complete a verified backup first, then touch the physical disk
Within 24 hours of the alert, finish "backup confirmed plus procurement-or-install decision" — no infinite observation
Label pulled drives with the failure symptoms and date; don't quietly recycle them into another machine

After boring patrol checks save you once, you'll appreciate how boring they always were. The goal of health monitoring isn't to manufacture anxiety; it's to make the replacement happen while you still have options.

One more word on procurement and spares: keep one cold spare of the same spec for common capacities so you're not waiting two days for shipping when it's time to swap. Power it up and run a short test occasionally, to avoid the black comedy of a "spare that was broken on arrival." Keep at least two alert channels (say IM plus email) — the night a single channel goes silent is often the night a disk starts coughing.