Replacing a drive

I’ll replace one of the 3TB drives with a 4TB drive. This’ll allow for a size upgrade (eventually–once all the drives are replaced with 4TB drives). It also means I’m rotating out older drives with new ones. I added a sticker to the drive to show the date, so future me can see which drives are oldest.

With the old drive removed, zpool status shows the pool as “degraded” with a drive missing.

NAME                     STATE     READ WRITE CKSUM
tank                     DEGRADED     0     0     0
  raidz2-0               DEGRADED     0     0     0
    ada0                 ONLINE       0     0     0
    ada1                 ONLINE       0     0     0
    ada2                 ONLINE       0     0     0
    ada3                 ONLINE       0     0     0
    ada5                 ONLINE       0     0     0
    9875896178717210589  UNAVAIL      0     0     0  was /dev/ada6

Plugging in the new drive makes no change here. Off to NAS4Free’s “Disk -> Management” screen. It shows a warning saying the physical devices have changed, and to import disks with the “clear configuration” option enabled. Do that, and Apply Changes. The disk is now listed normally, but with the Filesystem marked “unknown or unformatted”.

Now to the “ZFS -> Tools” screen. Select ‘replace a device’. Select the pool, tap next. Select “ada6” and tap next. It ran “zpool replace ‘tank’ ‘/dev/ada6′”, and now the status shows it silvering the new drive.

  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Jan 19 16:29:06 2018
        144G scanned out of 5.07T at 538M/s, 2h40m to go
        24.0G resilvered, 2.77% done

NAME                       STATE     READ WRITE CKSUM
tank                       DEGRADED     0     0     0
  raidz2-0                 DEGRADED     0     0     0
    ada0                   ONLINE       0     0     0
    ada1                   ONLINE       0     0     0
    ada2                   ONLINE       0     0     0
    ada3                   ONLINE       0     0     0
    ada5                   ONLINE       0     0     0
    replacing-5            UNAVAIL      0     0     0
      9875896178717210589  UNAVAIL      0     0     0  was /dev/ada6/old
      ada6                 ONLINE       0     0     0  (resilvering)

Hardware Installation

NAS Hardware Assembly

Building a NAS involves a few decisions followed by some lots of cabling. Choosing a good motherboard/case/power supply combo is an important first step. During the actual assembly you may find the manuals are not always easy to read, and that the cabling doesn’t always match. Below I show which cables go where and even which cables aren’t used at all.

Key Products

  • Motherboard: an ASRock Rack C2550D4I.

    This quad-core Intel Avoton board has plenty of performance for our needs, with a max of 64GB RAM, and 12 SATA connectors! One can think of Avoton as Xeon’s little brother – server-class and designed to go where a Xeon would be overkill. Avoton runs cooler, sips less energy, and costs less.
  • Case: Silverstone DS380B
    A small form factor case that can hold 12 drives? How is this even possible? Somehow they designed space for eight hot-swappable 3.5″ drives and an additional four 2.5″ drives!
  • Power supply: Silverstone ST30SF
    This is a properly-sized SFX unit supplying 300W, plenty for my 6-drive build.
  • Six 3.5″ drives


The Silverstone DS380 case is more compact that I had anticipated. I’m very pleased with both its capacity and diminutive small-form-factor proportions.

Case front

Case top and rear

The top fan is aided by two side fans, with a removable, cleanable, cover.

Side fan cover

The front has a power button and not much else. A door swings open to reveal the swappable 3.5″ drives. Also revealed are some audio and USB connectors that won’t be used for this build.

Door open

The drives are simple to pull out – just pinch the side tabs and pull.

Pulling drives

I’m using six drives, in a RAID-Z2 configuration. Photographing them like that is hard, so here they are mirrored:

Drives (mirrored)

To reach where the motherboard will be installed, access the case internals by removing the side panel. Two screws on the rear hold it in place (here they are shown partly unscrewed):

Side access screws

Four smaller black screws (on the side) hold the bay in place, and remember the two screws hidden on the case’s bottom!

Bottom screws

The 2.5″ bay is secured by two black screws on the side, and two silver screws on the rear.

You can see the black screws and the completed jumble of cabling:

Wiring jumble

Once the bays have been removed, you can install the power supply, RAM (onto the motherboard), and the motherboard into the case. There are several cables that won’t get used, e.g. GPU power, and those front USB ports whose cable has no matching connector on the motherboard.

These cables look important but aren’t used:

Unused cables

At 100mm deep, the power supply has enough space even with everything assembled.

Power supply squeeze

The power supply is simple enough to install, with four external screws and plenty of room. Here it’s held with two thumb-screws and two regular screws. The other two thumbscrews I moved to holding the side panel in place, as I’m more likely to want to access that in the future than mess with the power supply.

Power supply rear

Power supply

The tricky part is getting the motherboard connected. The case has a mass of wires. The motherboard has a even huger mass of connectors. Where does everything plug in? First comes the unmistakable power cable. Note that this motherboard can accept a larger style plug. Here you can see which pins are unused using this more compact plug.

Motherboard power

Many SATA options are available with this motherboard. Four plugs are blue, representing slower SATA 2…Blue SATA

… but most are black and are SATA 3 offering 6GB/s transfer rate.

Black SATA

The rear fan connector is on the lower left of the motherboard.

Rear fan connector

The case connectors, for power, reset, and lights, are quite fiddly and easy to mix up. (Photographing them is also a challenge!) Click for a larger image, and check this carefully:

Case connectors

Once the motherboard is hooked up, it’s time to connect the drive bay. Two power connectors, and SATA for each drive. Use the right-most SATA connectors.

Drive bay connections

And the side fan connectors, upper and lower:

Fan connectors

Finally, screw the drive bays into place (remember the bottom screws again!) and push drives into place (once they’ve been mounted into their sleds).

The motherboard offers three ethernet sockets. Ignore the one labelled MGMT (it’s for IPMI); use either of the other two. These USB ports are active and can be used to install your OS, along with the old-school VGA monitor and a USB keyboard.

External connectors

Plug into power, remember to switch the power supply’s button to the on position, and press the big round button on the case’s front. Time to install your OS of choice!

Products Used

All drives are 3TB capacity, for 12GB total in RAID-Z2.

Testing Bit Rot

One of ZFS’ most compelling features is detection of bit rot. Though rare, rot is almost impossible to detect until well after the event occurs and rotating backups have promulgated the error.

One ends up with photos where the top half is fine but the bottom is semi-random junk. Or glitches in home movies.

ZFS checksums everything – your data, its own data – and verifies the data on disk matches the checksum every access.

If you have mirroring or RAIDZ, then not only can ZFS tell you about the error, it can pull the correct data from a good disk and overwrite the bad data.

Bit rot, or data degradation, happens on hard drives via humidity, cosmic rays (we are bathed in the light of 150 million stars, one of them quite close), and loss of magnetism. Hard drives have a limited life time and individual disk sectors can go bad before the entire drive dies. Manufacturer specifications give error rates for their drives. This might be one in several terabytes, but nowadays terabyte drives are common; multiply the rate by your amount of data, give it enough time, and sooner or later you will experience it. ZFS to the rescue!

The Set Up

Let’s create a zpool, RAIDZ1, from 3 files.

truncate -s 200M file1
truncate -s 200M file2
truncate -s 200M file3
sudo zpool create mypool raidz1 \
  /home/graham/Documents/test/file1 \
  /home/graham/Documents/test/file2 \
sudo chown -R graham:graham /mypool

Full paths to the files must be given to zpool create, and we need to grant ownership to the pool so we can create files in it.

We can check the zpool is healthy:

$ sudo zpool status 
  pool: mypool
 state: ONLINE
  scan: none requested

	NAME                                   STATE     READ WRITE CKSUM
	mypool                                 ONLINE       0     0     0
	  raidz1-0                             ONLINE       0     0     0
	    /home/graham/Documents/test/file1  ONLINE       0     0     0
	    /home/graham/Documents/test/file2  ONLINE       0     0     0
	    /home/graham/Documents/test/file3  ONLINE       0     0     0

errors: No known data errors

The Test

Let’s create a small text file:

echo abcdefghijk > /mypool/a.txt

That gives us text that’s easily found in the pool’s data files:

$ grep abcde file*
Binary file file1 matches
Binary file file3 matches

ZFS is storing the data in 2 of the 3 storage devices, because we’re using RAIDZ1, which offers one mirror for all data.

Cosmic Ray Simulation!

We can flip some bits if we first offline one of the drives:

sudo zpool offline mypool /home/graham/Documents/test/file1

Now load up file1 in wxHexEditor, press Ctrl-F, and search for abcd.

Before Edit

Type over abcd to change it to anything you like.

After edit

You’ll need to change the file mode to “Overwrite”, so you can then File->Save.

Once saved, file1 no longer contains the correct data:

$ grep abcde file*
Binary file file3 matches

Online file1 again, so the zpool is whole:

sudo zpool online mypool /home/graham/Documents/test/file1

Results – Catastrophe?

ZFS hasn’t accessed any data yet, so doesn’t yet realize the error:

$ sudo zpool status   pool: mypool
 state: ONLINE
  scan: resilvered 2.50K in 0h0m with 0 errors on Sun May 24 10:30:30 2015

	NAME                                   STATE     READ WRITE CKSUM
	mypool                                 ONLINE       0     0     0
	  raidz1-0                             ONLINE       0     0     0
	    /home/graham/Documents/test/file1  ONLINE       0     0     0
	    /home/graham/Documents/test/file2  ONLINE       0     0     0
	    /home/graham/Documents/test/file3  ONLINE       0     0     0

errors: No known data errors

Though we introduced a data error, ZFS gives good data:

$ cat /mypool/a.txt 

No sign of any error! In my case, a status report still shows no errors, perhaps because the data came from a cache or because it read the data from file3, which we didn’t change.
We can force ZFS to check integrity with the scrub command:

$ sudo zpool scrub mypool 
$ sudo zpool status 
  pool: mypool
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
  scan: scrub repaired 512 in 0h0m with 0 errors on Sun May 24 10:34:42 2015

	NAME                                   STATE     READ WRITE CKSUM
	mypool                                 ONLINE       0     0     0
	  raidz1-0                             ONLINE       0     0     0
	    /home/graham/Documents/test/file1  ONLINE       0     0     1
	    /home/graham/Documents/test/file2  ONLINE       0     0     0
	    /home/graham/Documents/test/file3  ONLINE       0     0     0

errors: No known data errors

ZFS has certainly sat up and noticed now!

Note the changed status and the ‘1’ in the ‘CKSUM’ column for our cosmically zapped file1. ZFS wrote good data over bad (“scrub repaired”):

$ grep abcde file*
Binary file file1 matches
Binary file file3 matches

“Applications are unaffected.” Music to my ears – thanks to mirroring we had redundancy and ZFS was able to recover our data.

We can safely clear the error. If these were real disks, we would now decide whether to replace the potentially failing drive.

Catastrophe avoided!

Experimenting with ZFS Failures

While waiting for all the drives to arrive, I built a 3-disk RAIDZ1 configuration to perform tests on. Each of the 3 drives has a capacity of 3TB. RAIDZ1 means one of the disks is used for redundancy; instead of 9TB of storage space, there’s only 6TB. For that capacity loss we gain resilience to any one of the drives failing. If a drive were to fail, we could simply replace it and ZFS would continue like nothing had happened! Let’s try some experiments and see how that works.

Working Configuration

These shots from NAS4Free show the three disks, configured into one Virtual Device (vdev), inside one pool; the pool has one dataset.

Three disks:


Inside one pool:


The pool has one “virtual device”, a RAID-Z1.


The disks are bound into one dataset in the pool.

To begin with, the pool is ‘ONLINE’ and all three disks are working fine.

Unplugging cables

Let’s simulate a drive going bad; what happens if we unplug the SATA cable from one of the drives?


The pool is now DEGRADED and one of the drives is marked UNAVAIL. Uh oh! ZFS tells us to use zpool online to bring the drive back.

Even in this degraded state, I’m able to access my data – in fact our home folder (~) is located on this dataset, and operating perfectly. Even with only 2 out of 3 drives running. (If we were to lose a second drive, our data would be inaccessible).
Let’s re-attach the 3rd drive’s SATA cable and tell ZFS to online the drive.

It all works great; I wonder why it says it “resilvered 68K”. There are several MB of data in the pool. 68K is perhaps just some metadata.

Moving cables

How about we unplug a SATA cable from the motherboard and connect it to a different SATA port?

ZFS didn’t blink an eye; we didn’t even have to online the drive.

Let’s pretend the entire motherboard needed replacing, and we forgot which drives were plugged in where. I shutdown; unplug all the SATA connectors and re-connect them in different connectors. Power on and what happens?

Switching connections around made no difference at all! If that had been a real motherboard replacement I would have had not had to worry where the drives were connected.

New boot drive

Things get more interesting here. I’ve been booting this box off a 16GB USB flash drive. What if this drive went bad?

The boot drive contains the ZFS configuration; losing that means the fresh NAS4Free installation will need to discover what state ZFS and the drives were in. NAS4Free does recommend backing up your configuration, but let’s say you forgot to…

I installed NAS4Free on a new 8GB flash drive. I told it to configure the network card (option 2 on the main menu) and then visited the displayed IP address from my Mac’s browser.

Uh oh. Nothing here in ZFS-land! No pool, no disks!

Here’s the configuration screen, with no pool but a useful button labeled “Import on-disk ZFS config”. Later we can see what ZFS commands the button runs.

After clicking that – look what happened!

The ZFS pool, vdev, and dataset are back! While the Pools and Datasets web pages still show nothing, we can fix that, too – read on.
At this point I enabled SSH access so I could have a poke around and try to access some data. I enabled root SSH access, and was able to navigate to the ZFS dataset directory!

zpool status shows the pool as ONLINE.

It’s educational to run zfs history, which shows all the commands that were used to create the pool and also what command was executed when we imported after booting from the fresh USB drive:

Though NAS4Free’s WebGUI showed no pools or datasets, I was able to fix that using the ZFS -> Configuration page, which has a “Synchronize” button. After using that the rest of the WebGUI shows the pool and dataset correctly. This also fixed the Disks -> Management page, which had been showing no disks. As far as I can tell, that puts everything back the way it was (as far as ZFS goes – you may have had SAMBA shares etc too; so remember to backup your NAS4Free configuration!)


I simulated loss of a hard drive, loss of motherboard, and loss of boot USB drive. These simulations of failures turned out to all be recoverable situations! No data was lost at any step, which is great news for anyone with data they want to keep safe.

Note that even ZFS is not a substitute for backups – preferably off site, e.g. at a family member’s house or in a bank vault. An errant script or accidental manual file deletion means that ZFS will safely replicate that deletion across its RAID. ZFS snapshots could help here, but even so, your box is still vulnerable to flood, a lightning strike, power surge, or brown out which could damage one or all of the hardware components.
So far though I’m very pleased that ZFS, FreeBSD, and NAS4Free have lived up to their claims and provided a safe haven for my data!

I’ll be adding three more drives and setting up RAIDZ2. This will allow data access even with two drives gone. Research has shown that RAIDZ1 is not as safe an option as you might think – once one drive goes bad the odds of a second following it shoot up, and may not give you time to resilver a replacement disk.


General overview and advice for ZFS, Freenas, and configuration:

NAS4Free Hardware – 2015

Lately I’ve been building a ZFS-based NAS machine using the excellent NAS4Free. I’ve been wanting to use ZFS for years, and got rather excited when Apple was planing to include it with OS X years ago. Sadly that effort fell through (over licensing terms I’ve heard), but other OS’ have picked it up, notably FreeBSD. After an external drive died recently (fortunately in a dual-drive mirror, so no data was lost), I decided it was time to take the ZFS plunge. I updated the purchasing advice from DIY NAS 2015 Edition, going with Brian’s choice of:


Brian had trouble fitting his power supply into the case – his is 125mm long and SFX-sized power units are normally 100mm long. With a better fitting power supply:

  • I chose a Silverstone ST30SF, which has a lower wattage but I’m only planning on using six drives. It installs easily into the case.
  • Along with WD Reds and Seagates, I’m mixing in a pair of HGST Deskstar NAS drives, which are very well reviewed.
  • NAS distro: I tried FreeNAS but failed to get it to boot (! – probably user error), so switched to NAS4Free which worked first time, and has been rock-solid.

Items were purchased from,, Fry’s, and Best Buy. This was partly to obtain disks from separate manufacturing batches, so that failures would be unlikely to occur simultaneously.

This build has been rock-solid for several months now. I’m very happy with the chosen components.