EconTech

If you are a hard drive junkie, you have probably at least heard of the bit error rate. The last time I read about it was when I was happy to have these huge honking 40GB drives. Then, the error rate combined with the size of my drives, was small enough that it was not worth worrying about it. At some point, I forgot that the worry was actually a function of how much data there is, not to mention that the quantity of data is probably also a factor in the utility function for that data. Thus, as time has gone by and drive sizes increased dramatically, the bit error rate is of serious concern.

The short story first: Your consumer level 1TB SATA drive has a 44% chance that it can be completely read without any error. If you run a RAID setup, this is really bad news because it may prevent rebuilding an array in the case of disk failure, making your RAID not so Redundant.

The clincher is that this is easily fixed with a software update and a modest decrease in drive size to make space for error correcting data. This would increase drive prices modestly but would stave off lots of problems. Of course, explaining this all to the average consumer in two words or less (do you really look at much more than “750 GB” or whatever when choosing your HD?) is rather difficult.

The long story (from someone who knows much more about these things than myself):

SATA vs. SCSI reliability

Here’s a guy who discusses SATA vs. SCSI disk reliability. Short conclusion: actual disk failures (MTBF) are almost exactly as likely wtih cheap SATA disks as expensive SCSI disks. But the bit error rate of SATA is much higher. In other words, the likelihood of not being able to read a sector because it got corrupted on SATA is vastly higher than SCSI. By his calculation, on a 1TB disk, you have about a 56% chance of not being able to read every single sector, which means rebuilding your RAID correctly in the case of a failed disk is usually impossible.

It’s true, and I learned this the hard way myself. Back at NITI, we ran into exactly this problem. Back in the days when we introduced our software RAID support, typical disk sizes were around 20 GB, about 50x smaller than they are now. (Wow!) The bit error rates now are about the same as they were then, which means, assuming the failure percentage declines linearly⁽¹⁾, about a 1.1% failure rate in recovering a RAID.

In general, that 1.1% failure rate isn’t so bad. Remember, it’s 1.1% on top of the rather low chance that your RAID failed in the first place, and even then it doesn’t result in total data loss – just some inconvenience and the loss of a sector here or there. Anyway, the failure rate was small enough that nobody knew about it, including us. So when we had about a 1.1% rate of weird tech support cases involving RAID problems, we looked into it, but blamed it on bad luck with hard drive vendors.

By the time disks were 200GB and failure rates were more like 10%, we were having some long chats with those hard drive vendors. Um, guys? Your disks. They’re dropping dead at a pretty ridiculous pace, here.

You see, we were still proceeding under the assumption that IDE disk are either entirely good, or they’re bad. That is, if you get a bad sector on an IDE disk, it’s supposed to be the beginning of the end. That’s because modern disks have a spare sector remapping feature that’s supposed to automatically (and silently) stop using sectors when the disk finds that they’re bad. The problem, though, is it has to discover this at write time, not at read time. If you’re writing a sector, you can just read it back, make sure it was written correctly, and if not, write it to your spare sector area. But if you read it back and it fails the checksum – what then?

This is the “bit error rate” problem. It’s not nice to think about, but modern disks just plain lose data over time. You can write it one day, and read-verify it right afterwards without a problem, and then the data can be missing again tomorrow. Ugh.

And the frequency – per bit – with which this happens is the same as ever. With SCSI it’s less than with SATA, but as we have more bits per disk, the frequency per disk is getting ridiculous. A 56% chance that you can’t read all the data from a particular disk now.

There are two reasons you probably haven’t heard about this problem. First, you probably don’t run a RAID. Let’s face it, if your home disk has a terabyte of stuff on it, you just probably aren’t accessing all that data. Most of the files on your disk, you will probably never access again. Face it! It’s true. If you filled up a 1TB disk, you probably filled it with a bunch of movies, and most of those movies suck and you will never watch them again. Stasticially speaking, the part of your disk that loses data is probably in the movies that suck, not the movies that are good, simply because the vast majority of movies suck.

But if you’re using a RAID, you occasionally need to read the entire disk, so the system will find those bad sectors, even in files you don’t care about. Maybe the system will be smart enough not to report those bad sectors to you, but it’ll find them.

Secondly, even when you lose a sector here and there, you usually don’t even care. Movies, again: MPEG streams are designed to recover from occasional data loss, because they’re designed to be delivered over much less reliable streams than hard disks. What happens if you get a corrupt blob of data in your movie? A little sprinkle of digital junk on your screen. And within a second or so, the MPEG decoder hits another keyframe and the junk is gone. Whatever, just another decoder glitch, right? Maybe. Maybe not. But you don’t really care either way.

The Solution

At NITI, we eventually settled on a clever solution to this that won’t lose data on RAIDs. Of course we can’t protect data on a non-RAID disk in any direct sense, but we strongly recommended for our customers to do frequent incremental backups instead.

But on a RAID, the problem is actually easier: simply catch the problem before a disk fails. In the background, we would be constantly, but slowly, reading through the contents of all your disks, averaging about one pass per week. If your RAID is still intact but we find a bad sector, no data has been lost yet: the other disks in the RAID can still be used to reconstruct it. So that’s exactly what we would do! Reconstruct the bad sector, and write it back to the failing disk which could then automatically use its sector remapping code to make the bad sector disappear forever.

The read-reconstruction part was never open sourced, so if you want that, you’d have to write it yourself. Luckily, it was easy, and now that we have ionice you don’t have to be nearly as careful to do it slowly in the background.

The other part was to make sure Linux’s software RAID could recover in case it ran into individual bad sectors. You see, they made the same bad assumption that we did: if you get a bad sector, the disk is bad, so drop it out of the RAID right away and use the remaining good disks. The problem is that nowadays, every disk in the RAID is likely to have sector errors, so it will be impossible to rebuild the RAID under that assumption. Not only that, but throwing the disk out of the RAID is the worst thing you can do, because it prevents you from recovering the bad sectors on the other disks!

A co-worker of mine at the time, Peter Zion⁽²⁾, modified the Linux 2.4 RAID code to do something much smarter: it would keep a list of bad sectors it found on each disk, and simply choose to read that sector from the set of other disks whenever you tried to read it. Of course it would then report the problem back to userspace through a side channel, where we could report a warning about your disks being potentially bad, and accelerate the background auto-recovery process.

Sadly, while the code for this must be GPL as it modified the Linux kernel, the old svn repository at svn.nit.ca seems to have disappeared. I imagine it’s an accident, albeint a legally ambiguous one. But I can’t point you to it right now.

I also don’t know if the latest Linux RAID code is any smarter out of the box than it used to be. As we learned, it used to be pretty darn dumb. But I don’t have to know; I don’t work there anymore. Still, please feel free to let me know if you’ve got some interesting information about it.

Footnote

⁽¹⁾ Of course the failure rate is not exactly a linear function of the disk size, for simple reasons of probability. The probability of a 1TB disk having no errors (0.44, apparently) is actually the same as the probability that all of a set of 50x 20GB disks has no errors. The probability of no failures on any one disk is thus the 50th root of that, or 98.4%. In other words, the probability of failure was more like 1.6% back in the day, not 1.1%.

⁽²⁾ Peter is now a member of The Navarra Group, a software contracting group which may be able to solve your Linux kernel problems too. (Full disclosure: I’m an advisor and board member at Navarra.)

Further Reading:

• Ars Technica Forum Discussion (many other links here)