The email landed at 6:14 on a Tuesday morning, before I'd even had coffee. Subject line: Pool tank state is DEGRADED. My home TrueNAS box, a four-drive setup holding family photos, Plex media, and about a decade of backups, had lost a disk overnight.
I've fixed this exact situation more times than I'd like to admit, both on my own gear and for readers who email in a panic. So instead of a generic tutorial, let me walk you through the real recovery, start to finish, with the actual output I saw on screen.
The setup, and what actually failed
The machine was running TrueNAS SCALE 24.04 on an old HP microserver. Four 4TB WD Red drives in a single RAIDZ1 vdev, giving me roughly 10.9TB usable. Nothing fancy, and nothing enterprise about it.
The faulted drive was a WD40EFRX that had about 41,000 power-on hours on it. That's close to five years of continuous spinning, so honestly I was overdue for a replacement anyway.
RAIDZ1 tolerates exactly one drive failure. With one disk gone, my pool was still online and readable, but it had zero redundancy left. If a second drive burped during recovery, I'd lose everything.
Reading the zpool status output
First thing I did was drop into a shell. In TrueNAS you can do this from the web UI under System Settings, Shell, or over SSH. I ran zpool status tank and got this:
pool: tank
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
action: Online the device using 'zpool online' or replace the device.
scan: scrub repaired 0B in 04:12:33 with 0 errors
config:
Under the config block, one of the four drives showed FAULTED with a note reading "too many errors". The other three read ONLINE. That FAULTED line is the whole story: ZFS had kicked the drive out because it kept throwing read and write errors past its tolerance.
The numbers to the right of each device matter. ZFS lists READ, WRITE, and CKSUM (checksum) error counts. My bad drive had 38 read errors and a pile of checksum failures. The healthy drives were all zeros.
One detail worth understanding: a FAULTED state usually means the drive is genuinely dying, while an OFFLINE or UNAVAIL state can sometimes be a loose cable or a flaky SATA port. Mine was clearly the former, given the age and the SMART reallocated-sector count that had been creeping up for weeks.
Confirming which physical drive to pull
Here is where people make an expensive mistake. The pool refers to drives by GPTID, a long string like gptid/a1b2c3d4. You cannot walk to the server and pull "the second one" and hope.
I matched the GPTID to a real serial number with zpool status -v combined with the Storage dashboard in the TrueNAS UI, which lists each disk's serial. Then I used the drive identify light. If your chassis lacks one, run sas2ircu or simply note the serial and read it off the physical label before pulling.
Getting a replacement drive
I keep a cold spare on the shelf, which I strongly recommend for anyone running RAIDZ1. This time I had a fresh WD40EFPX (the newer CMR Red Plus) that cost me about 105 dollars. If you don't have one, order same-day or next-day, because your pool is naked until it arrives.
A note on drive choice: do not buy an SMR drive for a ZFS pool. SMR drives handle random writes terribly and can stall a resilver for days. Stick with CMR. Western Digital and Seagate both publish which of their models are which, and it's worth checking before you buy.
The replacement should be the same size or larger. Smaller will be rejected outright.
Running the replace and watching the resilver
With the new drive physically installed, I went to Storage, Manage Devices, clicked the faulted disk, and chose Replace. TrueNAS lists available unused disks; I picked the new one and confirmed. Under the hood this runs a zpool replace tank <old> <new>.
Immediately the pool state changed and the resilver kicked off. Back in the shell, zpool status now read:
state: DEGRADED
scan: resilver in progress since Tue Jul 1 06:52:10
1.84T scanned at 612M/s, 1.21T issued at 402M/s, 3.30T total
1.21T resilvered, 36.7% done, 01:58:22 to go
That "to go" estimate is wildly optimistic at first and settles down after ten minutes or so. My full resilver of about 3.3TB of actual data took just under four hours. As a rough rule, budget one to two hours per terabyte of used space on spinning disks.
| Pool state | What it means | Your action |
|---|---|---|
| ONLINE | All good, full redundancy | Nothing, relax |
| DEGRADED | Running but redundancy reduced or gone | Replace the faulted disk now |
| FAULTED (pool) | Too many devices lost, pool offline | Do not write; seek recovery help |
| RESILVERING | Rebuilding data onto a new disk | Wait, avoid heavy load |
What I did NOT do during the resilver
I did not start a scrub. A scrub on top of a resilver just fights for the same disk heads and slows everything down.
I did not copy new large files onto the pool, and I paused the nightly backup job that would have hammered it. Reads for Plex were fine, so the household barely noticed.
I also did not reboot. A reboot mid-resilver is usually survivable on modern ZFS, but it restarts the scan progress tracking and there's no reason to risk it.
The outcome
At 10:47 that morning, zpool status finally read what I wanted:
state: ONLINE
scan: resilvered 3.30T in 03:54:41 with 0 errors
config: all four devices ONLINE, 0 errors across the board
Zero data errors. Every file intact. The whole event, from panic email to fully healthy pool, took under five hours of wall-clock time and maybe twenty minutes of my actual hands-on attention.
Turning a scare into a habit
The reason this recovery was boring instead of terrifying comes down to three things I'd set up months earlier: email alerts were actually configured, I had a cold spare on the shelf, and I understood what the status output meant before I needed to.
If you take one thing from this: go check right now that your TrueNAS is emailing you on pool events, and put a spare drive in a drawer. For deeper reading on how drives fail and why redundancy matters, Backblaze publishes real-world failure data in their annual drive stats reports.
A degraded pool feels like an emergency. Treated calmly, with a spare in hand, it's really just a Tuesday chore.
Frequently asked questions
Is my data safe while the pool is degraded?
On RAIDZ1 with one drive faulted, your data is still fully readable because the parity covers a single failure. But you have zero redundancy left, so a second drive failing during recovery would lose the pool. Replace the disk as fast as you can and avoid unnecessary load.
How long does a ZFS resilver take?
It depends on how much actual data is stored, not the drive's full capacity. On spinning disks budget roughly one to two hours per terabyte of used space. My 3.3TB resilver on 4TB WD Reds took just under four hours.
Can I keep using the NAS during a resilver?
Yes, light reads like streaming a Plex movie are fine and the family barely noticed. Avoid large writes, pause scheduled backups, and do not start a scrub, since those compete for the same drive heads and slow the rebuild significantly.
What if a second drive fails during the resilver?
On RAIDZ1 a second failure during recovery means the pool is lost and you would be looking at professional data recovery or restoring from backup. This is exactly why RAIDZ2 or keeping current backups matters for anything you truly care about.
How do I know which physical drive to remove?
Match the GPTID shown in zpool status to the disk serial number listed in the TrueNAS Storage dashboard, then use the chassis identify light or read the serial off the physical label. Never pull a drive based on slot position alone.
Do I need the exact same drive model as a replacement?
No, but the replacement must be the same capacity or larger and should be a CMR drive, never SMR. SMR disks handle the random writes of a resilver poorly and can stall the rebuild for days.
