Hundreds of websites go titsup in Prime Hosting disk meltdown Hundreds of UK-hosted websites and email accounts fell offline when a disk array failed at web biz Prime Hosting. As many as 860 customers are still waiting for a fix more than 48 hours after the storage unit went titsup. The downtime at the Manchester-based hosting reseller began at 5am on 31 July, and two days later some …
"[P]romised that it has had a team working solidly for 36 hours without sleep in order to minimise the impact."
Knowing how goofy and error-prone I become after about 24 hours without sleep, that doesn't exactly fill me with feelings of confidence.
And it also gives me the impression that this is yet another company that thinks working people like pit-ponies is not only acceptable, it's laudable.
Some.
Drives do sometimes go in batches.
But if I was a customer I would want to be asked first before an old version of a site was brought on line - I mean there might be ordering systems with old pricing / stock figures or anything on there; in those circumstances better a holding page with an apology than a working site feeding through garbage.
Any server system with a smidgeon of professionality built into it will warn you when a drive becomes borderline.
Having 3 fail in one RAID6 array is... mindboggling...
Exactly how many drives do they have in each array, anyway?
Restoring old site backups to get the VMs up faster?
This is CRAP!
I'm guessing that what they brought back is the LAST FULL BACKUP of the failed array, and that they're now busy restoring Differential or Incrementals from after that.*
They should at least have the brains to keep the systems offline until they've restored everything, as it may otherwise result in lost orders and whatnot.
(What if someone browses to a webshop on one of those sites, orders something using a CC and they then restore over the transaction details? )
* Cheap bastards probably used incremental backups, too, instead of Differential, to save money...
Rather than jumping to the "This is all crap" conclusion, consider:
The array probably had all its drives purchased at the same time, this vastly increases the likelihood of drives failing in fairly quick succession.
If the array is new, it's entirely possible that the array that it was replacing is still kicking around, awaiting decommissioning. If it became apparent that the existing array is completely dead, it may have been a case of just zoning the old LUNs to the servers and away you go, with old data. This would also back up the first point. A recovery from tape could then take place to update the old data and the new array could be recommissioned when everything has settled down.
Error logs from array systems are there for a reason, which many unfortunately never bother to read.
With a 'new' system, that should be checked DAILY.
Automated emails from the system?
Sure, but I wouldn't trust them. Too many systems between the originator and me.
(sucks if the email warning of a problem with a RAID gets lost because the email storage is on the glitching array... Or, someone changes the IP of the SMTP server and the array box doesn't understand DNS. )
You really need to data-scrub, and watch the SMART statistics for the drives themselves, and act pro-actively. If the number of reallocated blocks starts increasing, replace that drive BEFORE the array is in peril. Sometimes drives do turn into bricks just like that, but in my experience and that of Google, an increasing rate of bad block reallocations after the array is first built is a warning not to be ignored.
If I were ever running a big-data centre, I'd insist on buying a few disk drives monthly, and from different manufacturers, so I could assemble new RAID arrays from disks no two of which were likely to be from the same manufacturing batch. A RAID-6 made out of drives with consecutive serial numbers is horribly vulnerable to all the drives containing the same faulty component that will fail within a month. I'd also want to burn in a new array for a month or longer before putting it into service. If a new drive is going to turn into a brick, it most commonly does so in its first few weeks (aka the bathtub curve).
"...If I were ever running a big-data centre, I'd insist on buying a few disk drives monthly, and from different manufacturers, so I could assemble new RAID arrays from disks no two of which were likely to be from the same manufacturing batch...."
It doesn't work like that, you get the disks you get when you buy an array. The array manufacturers spend ages testing that the firmware is compatible with the existing disks, that the disks are reliable and perform to spec, with the array and the array controller. There is far more chance of a failure caused by bad firmware or incompatibilities between disks and array/controller than mechanically. You will also be very hard pushed to find a array supplier who will support random disks being inserted into their array.
The best bet, is to have a healthy amount of online spares, for automatic rebuild and an array that phones home for more.
maybe it was hardware common to the disks, like a backplane or cable which failed.
Prime seem to have had a problem with their SAN for a while - it's been blamed for some slow (to non existent) server responses over the past couple of weeks. I'm not even sure it's a reliable design of SAN (I believe it was 'home baked' from what support staff told me <last time this happened!>).
Not much surprises me with Prime any longer, they don't usually appear to show a great deal of care or understanding about the importance of DNS, data or, come to it, their customers.
Not very mindboggling here: we lost a very large number of drives simultaneously when some muppet contractor managed to set off the fire suppression system whilst attempting routine "maintenance". Service induced failure is a lot more common than you might think in all sorts of areas.
Most likely they had an old copy of the data sitting on disk storage somewhere and brought that back on line as a quick fix whilst the tape system is recovering the backups. I can't remember the last time I saw a full backup done or imagine quite how long it would take. These days all our stuff is incremental into a library, and we just restore from the library without having to worry about when individual files were backed up.
Nothing can protect you against 'Acts of Dead Meat', unfortunately...
(Well, mirroring the array to another similar box in another location might... )
Full backups are important. They really are.
If for nothing else, they're really handy for off-site storage...
(To protect against flooding, fire, sabotage, theft... )
Replication is great, but like RAID, not a panacea.
Also, it's highly likely that the customers don't want to pay for that level of data security, it's very expensive, more than double the costs because you have to pay for the datalink as well as the extra servers and disk.
Perhaps it did!!!
(Now to get my damagement bashing in; and this is just speculation, mind you.)
Perhaps the warning signs were there, but damagement, in its quest for ever increasing profits, decided to hold off replacing the drives. Could it be that they did not want their quarterly bonuses to take a `hit`??? The spreadsheet jockeys could not find a line item for replacement drives.
Icon that says it all.
To paraphrase Oscar Wilde...
"To lose one disk, Mr Smith, may be regarded as a misfortune; to lose three looks like carelessness."?
After 3 disk failures I'd be checking the RAID controllers and stuff to ensure it's not something other than a disk issue. Unless you tell me it's software RAID that is, then I'll just laugh at you.
Nope. Disk #1 fails. A new disk is insterted and the array starts to rebuild. The act of rebuilding stresses the living shit out of the other disks, including accessing areas of them that haven't been looked at since Jesus was a lad[1] (e.g. parity stripes for O/S files on the failed disk that were written during a server installation early in the array's life and never touched since). Disks #2 and #3 turn their toes....
Of the three disks I have had fail in my own gear, two failed during full backup cycles and one in a RAID rebuild. It's heavy use of the entire disk that shines a glaring light on problems. This is also why anyone relying on incremental backups and thus not ensuring that the entire disk structure is kosher on a regular basis is asking for it.
Mixing batches of disks is unlikely to help, except in the unlikely case where a particular batch has a manufacturing defect. In such cases, they'll usually start dropping like flies at commissioning time anyway. What will help is ensuring that your RAID array is populated with disks with significantly different numbers of service hours on them, but since arrays tend to be commissioned in one go with new disks, this very rarely happens.
[1] This is why monitoring the SMART stats makes no odds. SMART only records errors when they are seen in normal operation, it does not proactively scan the entire surface looking for 'em.
Raid error reporting is one thing but if the raid card itself goes there is not even a hint of the impending doom.
Can't help thinking RAID is another one of those "Many beasts with one name" technologies that would benefit from some rigorous standards.
Drives from one controller will often not talk to later versions of the same controller (or have I just been unlucky)
AC just because many people in IT seem to think they have it all covered and "unknown unknowns" could never happen to them, pointing fingers and being smart may distract us from the discipline required.
We had a raid controller from Dhell do this - it went to write through mode as it should if it encounters errors; except instead of actually writing the data though (abet slowly) it decided any writes could be silently ignored and dropped.
People saying H/W raid is better than software raid are either never dealt with dodgy raid controllers; or are thinking of that joke of raid that comes built into motherboards and not mdadm.
Really? In my experience, people who say that software RAID is better than hardware RAID are OS engineers, who think that they somehow automatically know about either local or SAN attached storage infrastructure.
Software RAID, after all, still goes through disk controller chips, often the same one for multiple drives.
Drives from one controller will often not talk to later versions of the same controller (or have I just been unlucky)
No, thats one of the several reasons that these days I refuse to countenance hardware RAID controllers.
Another is the case where the manufacturer of your RAID controller goes out of business and the only place you can get a (maybe!) compatible replacement is E-bay. And then there's the time you find out the hard way that if you swap two drives by mistake, it immediately scrambles all your data beyond retrieval. And if there's a hardware RAID card that uses ECC RAM, I've yet to see it.
Use Linux software RAID. Modern CPUs can crunch XORs on one out of four or more cores much faster than SATA drives can deliver data. And auto-assembly from shuffled drives does work! You do of course have a UPS, and you have of course tested that UPS-initiated low-battery shutdown does actually work before putting it in production.
(Enterprise RAID systems with sixteen-up drives may be less bad, and in any case it's a bit hard to interface more than 12 drives to a regular server PC. It's little 4-8 drive hardware RAID controllers that I won't touch with a bargepole.
@Nigel 11 - I think we're talking about significantly different systems here. To me a RAID array is something that is free standing and has hundreds of disks. A the only locally attached arrays that I've used recently are made by HP (nee Compaq, nee DEC) and are 2u racks full of 2.4" disks, with controllers that have battery backed write cache - with error correcting RAM.
If your primary concern when buying an array is "will this company go bust", don't buy it. However, rest assured a proper, enterprise (or SME) class RAID controller/Array is way faster and more reliable than software, it also won't knacker your disks if you put them in, in the wrong order. It certainly won't loose writes if cached, when there's a power failure, which software RAID will.
"Prime Hosting has apologised to punters"
Um... no, no they have not. Not a single apology.
Their site has no information about downtime on it anywhere, their ticket support system is being completely ignored, their phone lines (which might be back up now) were down for all of yesterday with a recorded message basically saying "We know there issues, go away".
Their Twitter feed is the only thing with any information on, and that is remarkably lacking.
>their phone lines (which might be back up now) were down for all of yesterday with a recorded message basically saying "We know there issues, go away"
And you think calling them is going to make them suddenly get things back to normal?
If there is information on twitter then I would assume that is the current state of the problem, if you don't think so then complain later..
What do you want? A bit by bit commentary on twitter and a fully manned telephone ops room or maybe a dedicated line for you to constantly ask "what's going on? when will my really, really important web pages be available, don't you know there are poeole out their who haven't seen a pictrue of my pussy for more than ten minutes?", while one guy tries to get the thing back to its previous state?
Those are some nice logical leaps you've made there. True genius in the works.
Actually, despite you trying to make it sound completely ridiculous, a bit by bit commentary isn't exactly out of the question. It's not that unheard of for there to be people employed by a company that aren't experts in data recovery and server migration. Maybe those people not involved in that side of the issue could take a few minutes to at least to keep some worried customers updated?
Regardless, what I'd expect is the bare minimum of customer support:
1) At least one mention that there is a known issue on their website;
2) For their single point of support to be working (their ticket system was offline all day ). These shared servers that are down aren't their only hosting business.
3) Less than 9 hours between Twitter posts on the day the majority of sites went down.
4) To maybe ask customers whether they would like an unusably old backup in place before doing so;
5) To maybe let customers who do have an already unusably old backup in place that they can provide a more recent one, so there's not a need to panic (12+ hours between putting some backups in place and then sending out a Tweet).
That is not much to ask, seeing as they have just destroyed a good number of businesses (not mine, chillax before you start worrying about my transexual cat photo enterprise).
One of those techs people think is the holy grail to save you having to spend lots of dosh on a duplicate or clustered system. If anyone wants to save themselves from getting into the hell hole of a situation; always have backup on backups and duplicate on duplicate systems.
Where I'm working at the moment; we have 6 duplicate servers hosting all our websites and all the elements and database hosted on a clustered group of servers. Even the local hard-disks on the servers are RAIDed for performance reasons on top of availability. It would take a lot to go wrong for us to go fully tits up.
Whilst looking in vain on their website for any scrap of information as to what the fudge had happened to my sites, I enjoyed* the irony of finding this press release:
http://www.primehosting.co.uk/news/Recruiting_Again
How nice of them, I thought, to be blowing their own trumpets whilst quite literally in the middle of the biggest clusterfudge a hosting company could hope to experience. Surely, I wondered, the PR person pimping this press release could instead be informing customers as to when their sites might re-appear? But apparently not.
* did not enjoy
This post has been deleted by its author
Do you take regular backups of your sites?
I certainly do and can restore the lot reasonably quickly. I have tested that too.
This article does present a scenario I hadn't thought of though, namely that of the ISP restoring older backups over whatever I might have already restored.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
