9

I am a web developer. I have not much experience in hardware. For this reason, I use managed servers.

This morning, one of the drives in our setup failed. However, the full site went down. I asked my web host what happened and he replied that the hard disk failed in such a way that the RAID controller couldn't work properly. The array was set up as RAID 4.

Do you guys ever seen that before? Is it possible?

Thanks for any help on this guys. I need to know if my web host is honest with me.

user
  • 4,445
Steve Rodrigue
  • 99
  • If more than one disk in an Array dies then the RAID will fail (though it depends on the RAID setup). – Rhys Evans Nov 26 '12 at 15:55
  • Short story is, your provider is an a****** and works on the cheap side. That could be perfectly acceptable as long as you as a customer have been warned that his infrastructure is not fault-tolerant do drive failures. – Luke404 Nov 27 '12 at 11:47
  • Please update the question with the raid type (i.e. raid 0,1,4,5,6,etc). – Trevor Boyd Smith Nov 27 '12 at 13:43

7 Answers7

22

More likely than not your provider is using Hard Drives that are not meant to be used in RAID. Normal consumer SATA drives fall into this category.

The likely problem is that the drive started experiencing Uncorrectable Read Errors (UREs). When this happens in a consumer drive, the drive sits there and retries the read operation (usually for 30-60 seconds) until it gives up. The RAID will wait for the drive to report the error (the 30-60) seconds. So a simple request for a few sectors can easily cause the server to grind to a halt while the failed drive grinds through those read-retry operations.

Drives that are meant for RAID Arrays have either Time Limited Error Recovery (for SATA drives). TLER reports failures back to controllers quickly, so that the controller can intelligently respond to such failures (mostly intelligently; hopefully). SCSI (SAS too) work somewhat differently. The SCSI command set allows the controller to specify various recovery effort limits on drives (MODE SELECT: RW ERR RECOVERY). A RAID controller should set the drives to fail quickly, the controller can then test if the drive thinks that it's working properly with the TUR command, fail the drive out of the array if there's a check condition.

Chris S
  • 78,185
11

Yes, this is possible, even in scenarios where you would think the array should have survived the failure.

Some possibilities as to why an array fails:

  • More drives failed than could be sustained by the RAID mode. For example:
    • RAID 0 (striping) can not survive any drive failures.
    • RAID 1 can survive failures of all but 1 drive.
    • RAID 4/5 can survive 1 drive failure.
    • RAID 6 can survive 2 drive failures.
    • RAID 10 can survive the failure of up to 50% of the drives, depending on which drives fail.
  • A bug in the RAID software or controller firmware.
  • User error.
    • Someone pulled too many drives.
    • Someone pulled a drive and never replaced it, and another drive subsequently failed.
    • The array was not monitored, allowing more drives to fail than could be survived.
  • Cheap controllers with consumer grade drives are commonly known to fail even in otherwise survivable scenarios.
    • A consumer level drive will attempt almost indefinitely to read a bad sector until it gets a good read. A cheap controller will wait almost indefinitely for such a drive to return a result. The wait can be so long that the operating system gives up. Then on reboot the drives don't respond quickly enough to the controller and the array is assumed to be failed.
    • On the other hand, an enterprise level drive will give up quickly, allowing the controller to pull the data from another drive. Also, a good controller will mark a drive that takes too long to respond as failed and move on.
longneck
  • 23,202
  • 1
    RAID 1 should survive the death of all but one of the drives in the array. Granted, most people probably run two-drive RAID 1 setups, which means that it can only survive the death of a single drive, but that is not inherent to RAID 1. – user Nov 27 '12 at 10:47
  • Interesting so if 1 disk in a RAID 10 fails you should break another disk because it wont survive if only one disk is broken :-) I think you should edit your post. – FLY Nov 27 '12 at 12:50
  • @MichaelKjörling good point. I edited my post. – longneck Nov 27 '12 at 13:44
  • @FLY you're right, i glossed over that point. edited. – longneck Nov 27 '12 at 13:45
  • RAID4 should be RAID3. RAID3 is byte striping with parity; RAID4 was an ECC implementation needing a huge number of drives that AFAIK was never implemented. – Dan Is Fiddling By Firelight Nov 27 '12 at 14:56
  • @DanNeely no, your are incorrect. both RAID3 and RAID4 use one dedicated parity drive. RAID3 is byte striping, RAID4 is block striping. i only mentioned RAID4 because the OP said that is what is in use in his situation. – longneck Nov 27 '12 at 15:03
  • @longneck You're right; I was thinking of raid 2. – Dan Is Fiddling By Firelight Nov 27 '12 at 15:16
8

If it was a RAID 0 implementation then certainly when a single drive fails you'll lose the array and all of the data with it.

joeqwerty
  • 110,860
  • It's a RAID 4 implementation – Steve Rodrigue Nov 26 '12 at 15:59
  • 11
    hahaha - you nearly had me there, what is it really? – Chopper3 Nov 26 '12 at 16:04
  • @SteveRodrigue Sorry? – Dan Nov 26 '12 at 16:04
  • 3
    @Chopper3 NetApp uses RAID4. So it's not completely unheard of, though it did give me a chuckle too. Maybe that's the host's way of saying they have a NetApp Filer or something. – HopelessN00b Nov 26 '12 at 16:09
  • 1
    @SteveRodrigue Are you sure it's RAID 4? – MDMarra Nov 26 '12 at 16:24
  • 1
    If it is indeed RAID4 and only 1 drive has failed, then it ought to be possible to install a new drive and rebuild the array, in principle at least. Perhaps the web host meant that one of the remaining drives failed while he was trying to do this? – user3490 Nov 26 '12 at 19:11
  • I've seen a server with RAID-1 crash before when a single drive failed. Why? I couldn't tell you, but it does indeed seem to be possible with lower-end RAID controllers. What controller is on the server? – Bigbio2002 Nov 26 '12 at 20:07
2

I have seen firmware bugs take out the whole RAID when a disk goes bad, or when it starts reporting imminent failure. Sorry, I have nothing specific to point you at, but yes, it can happen. Not as part of the RAID spec, of course, it is definitely a bug.

chutz
  • 7,990
1

Yes, it's possible. It's not supposed to happen, but it certainly can. Enter UREs ( Unrecoverable Read Error) and controller faults and firmware bugs and the like.

Without additional information (that your host probably won't give you), it's not possible to say definitely one way or the other, but anyone who's worked with a lot of RAID arrays has had experiences where a whole array was lost or crashed when it shouldn't have.

(And, by the way, RAID4 isn't a very commonly used RAID level, but should withstand the loss of any drive. Doesn't mean it always will, however.)

HopelessN00b
  • 53,954
1

I've had many HDD failures where not the mechanics failed, but the electronics making up the communications interface. Because of their small size many electronics components are very sensitive to even minor electrical irregularities (this can happen when large A/C motors nearby are turned on/off etc. and the power supply is a little on the cheap side).

When the drive's internal power converters or capacitors (energy storage buffers) burn out the electrical signals generated at the HDD's external connectors can and will move way out of specification. Since the drive is connected to the controller via copper wires, and often in servers many drives share a cable connection to ease installation and reduce clutter, this can easily disrupt or even permanently destroy any number of adjacent components.

This has very little to do with pricing by the way. It is true that expensive controllers and drives MAY use parts that are more tolerant to abnormal conditions or have better shielding, and that with budget components you are more likely to get sub-standard parts. But I have regularly found identical capacitors on a $50 drive and a $500 drive. And if a failed HDD directly routes 12 Volts from the power supply to the SATA connector because something shorted out, your RAID controller will be fried, no matter how many figures the price tag had.

It's not what usually happens, but it's definitely not unheard of in my experience.

Jost
  • 31
  • "often in servers many drives share a cable connection" Not in modern SAS or SATA environments. It's pretty astronomically unlikely that your scenario is what happened here; I don't think I've ever heard of a drive's electronics dying and taking other components with it. While 12v would certainly fry a SATA or SAS controller, the logic components are very rarely connected to the 12v in any way, as stepping the voltage down from 12 to 3.3 or less is very complicated compared to 5v or 3.3v sources. I'm curious where you might have head of this sort of thing happening; if you're willing to share? – Chris S Nov 27 '12 at 02:49
1

Yes, I guess the whole raid can fail after a single drive failure. The first failing drive will be taken offline by the controller and the raid will still work fine. But when the failed drive is replaced, the controller starts rebuilding the raid. If there is a latent non-discovered read-problem on one of the other remaining drives, a rebuild of the failed drive could cause more drives to go offline (when read-problems are discovered while rebuilding the raid) again causing the whole raid to fail.

Pytagoras
  • 11