Re: Hooray for single points of failure!
Well, in my environment, the critical data is -also- stored on large central arrays. The model I've gone for is to break tasks up into as small of pieces as possible and then distribute the tasks across as many boxes as needed. We prefer scaling out rather than up.
With our mission-critical system, rather than scaling up, we scale out. So rather than using a high-end 8-way Xeon E7 system with a 1+ TB of RAM and a 24-disk bay filled with high-end SSDs, we use 32x Xeon E3 machines with 64 GB of RAM and some mid-range SSDs. For databases, we split the tables so that very active and CPU-intensive end up on many of the systems, even mostly inactive tables end up on several machines.
We found that its very cost-effective to just distribute the work load to many mid-range servers. When pushing a system further and further, it was found that at a certain point each additional FLOP / IOP we squeeze out of system, the more expensive it was, so we build systems up until we get to the point where an additional 10% performance is going to push costs up further than 20%.
We also save a lot of money on staffing as well. Rather than having staff working around the clock, we just have a skeleton crew outside of business hours whose task is just to diagnose why something failed rather than attempting to repair: When you have a pair of large system in an HA configuration, you need to repair them quickly to ensure things run smoothly; but when 1 out of 32 servers go down, capacity is only reduced 2-3% and can be handled on Monday when normal staff come in. We also saw a lot of gains in routine maintenance, we don't need to wait until after-hours to patch a system, we can do it any time and no one outside of IT even notices that something was down.
We even do that model with our file servers, we use these 1U boxes with 12x 3.5" disks in them and filled with 6-8 TB drives. With everything replicated and advanced load-balancing, we can squeeze many more IOPs out of a rack of boxes with inexpensive spinning-rust and 1-gig interfaces than the 1.5-rack monstrosity EMC sent us to try out.
The developers we use for in-house development are highly-disciplined and code is profiled quite regularly so that we can identify inefficient databases, overloaded functions, or just inefficient methods. Our head of development is a firm believer in the Unix model of building things so they do only one thing but does it well and does it quickly. Our code may be a bit large, but the systems zip right through it without breaking a sweat and its very, very easy to understand and debug. Managed to reduce development staff from 150+ devs in India and China to a dozen folk in Austin and another dozen in Frankfurt.