20

Not really familiar with Intel's "Mirrored Channel Mode" for a Blade Server setup (your typical moderately-heavy MySQL OLTP database running on the bare metal blade; no virtualization right now).

From the Intel docs I was able to find:

The Intel Xeon Processor 5500 series and Intel Xeon Processor 5600 series support channel mirroring to configure available channels of DDR3 DIMMs in the mirrored configuration. The mirrored configuration is a redundant image of the memory, and can continue to operate despite the presence of sporadic uncorrectable errors. Channel mirroring is a RAS feature in which two identical images of memory data are maintained, thus providing maximum redundancy.

On the Intel Xeon Processor 5500 series and Intel Xeon Processor 5600 series processors based Intel server boards, mirroring is achieved across channels. Active channels hold the primary image and the other channels hold the secondary image of the system memory. The integrated memory controller in the Intel Xeon Processor 5500 series and Intel Xeon Processor 5600 series processors alternates between both channels for read transactions. Write transactions are issued to both channels under normal circumstances.

However, I'm not really pickin' up what they're layin' down here. I lose half my storage capacity, but I gain "redundancy" of memory and possible gain read/write performance benefits? Like RAID 1 for RAM? Anybody have any practical experience with this configuration?

gravyface
  • 13,987

4 Answers4

8

"RAID 1 for RAM" is an accurate description. In my experience, there isn't much performance benefit, but depending on the bus speed vs the speed of the modules, your mileage may vary.

As far as redundancy goes.. well, it's not terribly often that a module goes bad.

Personally, I turn off mirroring whenever I see it enabled.

Shane Madden
  • 115,228
  • 2
    thanks Shane. Ever do any benchmarking before/after? – gravyface Mar 04 '11 at 19:32
  • @gravyface Can't say that I have, unfortunately; just haven't noticed a discernible performance different on vs off (on database servers and vm hosts). Some hard numbers would definitely be nice. – Shane Madden Mar 04 '11 at 19:38
  • 2
    I'm going to do some benchmarks then. See if it makes a difference. Can't say the error-correction benefit sounds like a tangible benefit, but I'm curious to see how it performs. I'm going to wait a couple days for some additional answers and then mark this correct. – gravyface Mar 04 '11 at 20:15
  • I see dimms go bad failrly regualarly, however given the size of the environment I should see 1 dimm every 2 weeks (statistically speaking) – Jim B Mar 04 '11 at 20:22
7

Personally I would sooner use some form of clustering rather than that level of hardware resilience. It makes sense for cheapo components like disks to double up on them, but mirroring memory is a nice-to-have but not that useful. I mean what's more likely to fail; a CPU, your OS, your software, your mobo, your PSU/s. I'd sooner put the money towards clustering.

Chopper3
  • 101,651
  • 1
    My thoughts exactly: while certainly useful for a very specific failure point, I could tick off many more HA "boxes" by going the clustering route with another blade or two (and likely another chassis in another data center). – gravyface Mar 05 '11 at 03:33
  • 1
    clustering won't help you when calculations have to be done on schedule or within a tineframe. In some situations failover takes longer than the operation required to be performed – Jim B Mar 06 '11 at 01:33
5

I have read that this kind of thing (you can do it with CPUs as well) is very useful in the huge supercomputer clusters.

Some of these clusters are running so many machines that there will be a machine failure every couple of hours. Faster than the jobs can complete. That really messes up the computation. Adding redundancy like this to each node can more than double the time between failure.

Zan Lynx
  • 906
  • so this is high-end stuff now trickling down to the mainstream, I take it. Not really seeing much value for my needs. Thanks though. – gravyface Mar 05 '11 at 03:29
  • Yes, it is high end. Wait until you get hot plug CPU's AND CPU MIRRORING (!) in a pc ;) Mainframes can switch over to another cpu when one fails. – TomTom Mar 05 '11 at 11:45
4

This memory mode was really designed for situations where you need high availability You shouldn't see much of a performance difference (since the loss of one channel probably isn't noticeable under normal operations) however you actually lose alot of ram. With mirroring enabled, only one-third of total memory is available for use because two DIMM slots are the primary channel, two DIMM slots are the backup channel, and two DIMM slots are not used. (at least that's how it is on IBMs)

I typically recommend that it be turned off (if you have an app or OS that likes ram - and let's face it: is there one that doesn't?) or save up to upgrade to the ex5 chipset from IBM (hp and others soon to follow with similar offerings) that adds a boatload more QPI.

There are the occasional "this server has to be up regardless of the number of shots fired at it" and this type of redundancy helps. Additionaly of you've purchased less than stellar quality ram this might save you from a blue screen or 2.

gravyface
  • 13,987
Jim B
  • 24,141
  • Yeah, don't see much need for this right now. Obviously nobody likes downtime, but when we address HA, we'll be looking at clustering most certainly. – gravyface Mar 05 '11 at 03:31
  • 1
    this in response to your "see one bad DIMM every two weeks" -- how often are you seeing bad DIMMs past the grace period? I can't recall ever having bad RAM once in production; I usually notice it the first few hours/days under typical workload. – gravyface Mar 05 '11 at 03:36
  • Our results mirror the rate google sees http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf. We have similarly configured servers- fully populated boards with lots of dimms, and applications that are memory intensive. In looking at 1 of my vmware environments I see 3 bad dimms across 18 fully populated IBM hs22vs (324 dimms) These servers have been up for about a year now. – Jim B Mar 06 '11 at 01:31