One of ZFS’ most compelling features is detection of bit rot. Though rare, rot is almost impossible to detect until well after the event occurs and rotating backups have promulgated the error.

One ends up with photos where the top half is fine but the bottom is semi-random junk. Or glitches in home movies.

ZFS checksums everything – your data, its own data – and verifies the data on disk matches the checksum every access.

If you have mirroring or RAIDZ, then not only can ZFS tell you about the error, it can pull the correct data from a good disk and overwrite the bad data.

Bit rot, or data degradation, happens on hard drives via humidity, cosmic rays (we are bathed in the light of 150 million stars, one of them quite close), and loss of magnetism. Hard drives have a limited life time and individual disk sectors can go bad before the entire drive dies. Manufacturer specifications give error rates for their drives. This might be one in several terabytes, but nowadays terabyte drives are common; multiply the rate by your amount of data, give it enough time, and sooner or later you will experience it. ZFS to the rescue!

The Set Up

• Linux Mint – I’m using version 17.
• ZFS-on-Linux installed.
• No need for special hardware – we’ll fake this all in the local filesystem.
• A binary file editor – I use wxHexEditor.

Let’s create a zpool, RAIDZ1, from 3 files.

truncate -s 200M file1
truncate -s 200M file2
truncate -s 200M file3
sudo zpool create mypool raidz1 \
  /home/graham/Documents/test/file1 \
  /home/graham/Documents/test/file2 \
  /home/graham/Documents/test/file3
sudo chown -R graham:graham /mypool

Full paths to the files must be given to zpool create, and we need to grant ownership to the pool so we can create files in it.

We can check the zpool is healthy:

$ sudo zpool status 
  pool: mypool
 state: ONLINE
  scan: none requested
config:

	NAME                                   STATE     READ WRITE CKSUM
	mypool                                 ONLINE       0     0     0
	  raidz1-0                             ONLINE       0     0     0
	    /home/graham/Documents/test/file1  ONLINE       0     0     0
	    /home/graham/Documents/test/file2  ONLINE       0     0     0
	    /home/graham/Documents/test/file3  ONLINE       0     0     0

errors: No known data errors

The Test

Let’s create a small text file:

echo abcdefghijk > /mypool/a.txt

That gives us text that’s easily found in the pool’s data files:

$ grep abcde file*
Binary file file1 matches
Binary file file3 matches

ZFS is storing the data in 2 of the 3 storage devices, because we’re using RAIDZ1, which offers one mirror for all data.

Cosmic Ray Simulation!

We can flip some bits if we first offline one of the drives:

sudo zpool offline mypool /home/graham/Documents/test/file1

Now load up file1 in wxHexEditor, press Ctrl-F, and search for abcd.

Type over abcd to change it to anything you like.

You’ll need to change the file mode to “Overwrite”, so you can then File->Save.

Once saved, file1 no longer contains the correct data:

$ grep abcde file*
Binary file file3 matches

Online file1 again, so the zpool is whole:

sudo zpool online mypool /home/graham/Documents/test/file1

Results – Catastrophe?

ZFS hasn’t accessed any data yet, so doesn’t yet realize the error:

$ sudo zpool status   pool: mypool
 state: ONLINE
  scan: resilvered 2.50K in 0h0m with 0 errors on Sun May 24 10:30:30 2015
config:

	NAME                                   STATE     READ WRITE CKSUM
	mypool                                 ONLINE       0     0     0
	  raidz1-0                             ONLINE       0     0     0
	    /home/graham/Documents/test/file1  ONLINE       0     0     0
	    /home/graham/Documents/test/file2  ONLINE       0     0     0
	    /home/graham/Documents/test/file3  ONLINE       0     0     0

errors: No known data errors

Though we introduced a data error, ZFS gives good data:

$ cat /mypool/a.txt 
abcdefghijk

No sign of any error! In my case, a status report still shows no errors, perhaps because the data came from a cache or because it read the data from file3, which we didn’t change.
We can force ZFS to check integrity with the scrub command:

$ sudo zpool scrub mypool 
$ sudo zpool status 
  pool: mypool
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: scrub repaired 512 in 0h0m with 0 errors on Sun May 24 10:34:42 2015
config:

	NAME                                   STATE     READ WRITE CKSUM
	mypool                                 ONLINE       0     0     0
	  raidz1-0                             ONLINE       0     0     0
	    /home/graham/Documents/test/file1  ONLINE       0     0     1
	    /home/graham/Documents/test/file2  ONLINE       0     0     0
	    /home/graham/Documents/test/file3  ONLINE       0     0     0

errors: No known data errors

ZFS has certainly sat up and noticed now!

Note the changed status and the ‘1’ in the ‘CKSUM’ column for our cosmically zapped file1. ZFS wrote good data over bad (“scrub repaired”):

$ grep abcde file*
Binary file file1 matches
Binary file file3 matches

“Applications are unaffected.” Music to my ears – thanks to mirroring we had redundancy and ZFS was able to recover our data.

We can safely clear the error. If these were real disks, we would now decide whether to replace the potentially failing drive.

Catastrophe avoided!