*Thunk* No worries, the UPS should spin up. Oh cool, it's in bypass mode Whatever can go wrong will go wrong. It's a law most IT people would understand and perhaps even fear. It was my third day as the new network manager for a reasonably sized estate across several sites, most inhabited by weirdy beardies who had jobs like counting bats, frogs and other animals you may never have heard of. It …
Sometimes it’s beancounters. More often, it’s not (at least in my experience). In a lot of cases it’s actually easy to get what you want, but it’s up to IT to get the message across. In this case it wouldn’t have helped of course, it’s what the previous guy should have done.
Get your disaster scenarios straight, and the options with detailed costs to mitigate. For each scenario, give an estimate of likelihood and potential cost to business (eg 1 site goes down, 100 users are twiddling thumbs for 2 days, that’s 200 days worth of wages, if it’s a warehouse add costs for late deliveries during peak season and so on and so forth). Put it all in a nice excel file with estimates from all and sundry. Sales & Marketing can be your friend here for the intangibles like reputation loss (they often tend to set their “disaster” cost estimates higher than they really are because their bonuses depend on it). If a customer has to wait longer for an order, in B2B environments for large orders and in B2C with social media shitstorms, the costs add up.
If you did all that and the beancounters say no, ask why and make sure they give you the numbers on which they base their conclusion (in writing). Because basically in that case they’re telling you they don’t want insurance for when the house burns down.
When disaster strikes, *then* you can blame them.
I’ve had my share of conflicts with them, but we’ve reached an equilibrum nowadays. I’ve cut down on maintenance contracts by stocking more spares which looks good on the balance sheet after year one, they are more easygoing when I want secondary connections to the main sites or more redundant power to the secondary data center. It takes time, but it’s worth it.
"give an estimate of likelihood and potential cost to business (eg 1 site goes down, 100 users are twiddling thumbs for 2 days, that’s 200 days worth of wages, if it’s a warehouse add costs for late deliveries during peak season and so on and so forth). Put it all in a nice excel file with estimates from all and sundry"
In my experience, unless your likelihood figures are realistic, you need to have a chance of failure >33% before anyone from beancounting and/or senior manglement will take the risk remotely seriously. I've seen plenty of people who should know better assume that a 10% chance is the same as "will never happen". Even if you've got accurate ways of measuring the risk (because a lot of this is frequently finger-in-the-air wizardry), too many people will ignore the risk entirely anyway, because those IT guys are always pessimistic and grumbling.
Of course, as Pterry so rightly pointed out, million to one chances come true nine times out of ten.
A previous job we had several racks in DR using only one UPS, and we said the chance of one of the UPS failing during an actual DR was about 20% due to the increased load (hand-wavery here was we didn't know what the actual max load was going to be during DR, nor exactly what the maximum load on the aged UPSs could withstand before falling over). Beancounters and senior management said that this was an acceptable failure risk and said that, if the UPS failed, just plug some or all of the servers into the UPS in the next rack to take the load off the overloaded UPS. This was added to the DR plan by the beancounters without knowledge or signoff from the techies, because who needs their opinion?
Guess what happened come the next DR (exercise thankfully, not an actual disaster). The business learnt the hard way that "cask aiding" doesn't mean assisting out a forlorn barrel that's down on his luck.
Of course, it turned out it was actually ITs fault for not correctly identifying the risk, because the chance of failure in hindsight was actually 100%, and the beancounters couldn't have been expected to allocate budget accordingly if IT gives incorrect information. If half the racks hadn't have died, the chance of failure would have been 0% and thus IT would have been at fault for incorrectly saying there was a risk of failure and the beancounters would have been entirely right in denying ITs frivolous request.
<need a Catch 22 icon>
Hey, beans don't count themselves, you know.
Beancounters are not your enemy. They've got a job to do, and it's a real (mostly boring, mostly thankless) job that needs doing.
Management, there's your enemy. Not your line manager, although they may become so if you don't cultivate them properly, but the real management. You know, the ones who take decisions about what risks are "acceptable" and what memos to ignore.
On a related note, another enemy is Chicken Little employees and consultants who send scaremongering memos about every conceivable risk, without properly quantifying it. When you tell the boss "a power cut will CRASH THE COMPANY", make sure you include quantitative assessment (likelihood per year of unscheduled power outage in this location, likelihood it will occur during business hours, and a specific projection of likely losses). The beancounters can actually help you with that: get them on your side.
And a proposal to mitigate the effect using a UPS, obviously, needs to include an allowance for maintenance of said UPS.
I left a few years later but the last I heard the company had spent several million pounds on a new site built directly on a flood plain with the IT hardware in the basement.
You'd be surprised about how many people don't even know what a floodplain IS, let alone the consequences of putting important infrastructure there. Even in the Netherlands (a country known for it's water management) some companies and homeowners have had to learn this lesson the hard way.
"You'd be surprised about how many people don't even know what a floodplain IS"
In the UK:
https://flood-map-for-planning.service.gov.uk/
https://www.gov.uk/check-flood-risk
In NL:
http://www.overstromingsrisicoatlas.nl/
https://www.risicokaart.nl/
www.overstroomik.nl/
"... some companies and homeowners have had to learn this lesson the hard way."
The very hard way as I think Dutch insurance doesn't cover flood risks. All the reason to pay the Waterschapsbelasting (dyke maintenance levy) diligently :)
The official monster raving loony party actually had a policy on floodplains.
Part of it says
Under a Loony government any prospective home purchaser be issued with a full description of such dictionary terms as ‘floodplain’, ‘coastal erosion’ and ‘exposed headland’. This will save time explaining why they have no house anymore after nature takes charge of the environment.
Read more at http://www.loonyparty.com/5908/3058/floods/ could be a useful idea for companies too.
Some of their policies are actually quite sensible often with something bizarre tacked on at the end to make it a bit more loony.
Any MP who’s constituency sells off a school playing field to developers will be required to relinquish his/her own back garden as a replacement sports facility for the school.
The Loonies propose that a minimum requirement of Maths ‘O’ Level be made for all government ministers and their treasury advisers, thereby preventing two different rates of inflation when used to calculate raises in both state benefits and taxes.
All third world debt will be cancelled. They’re not going to pay anyway. You know that. I know that. Don’t deny it.
Some of the policies have been enacted and some become law such as:
Passports for pets
Abolition of dog licences
Carnaby Street pedestrianisation
All day opening of pubs
Commercial Radio
How the hell does the monster raving loony party have a more sensible answer to this than the actual government?
They frequently do have more sensible solution. To the extent that an idea which first appears in an MRLP manifesto is later adopted by one or more of the "sensible" parties.
There are more ways of taking the piss out of conventional political parties than by standing candidates with silly names. Coming up with better policies is even funnier.
How the hell does the monster raving loony party have a more sensible answer to this than the actual government?
My 14-year-old got really interested in the last general election, having seen a party-political for the MRLP, and not just for the really odd policies like re-introducing mermaids. Put it this way, he's politically aware enough that he's currently wearing this T-shirt, and while he did disappear for a couple of minutes while we were queueing at the supermarket this morning, he came back with a copy of Private Eye (link included for non-UK readers who might not be aware of this publication).
There's hope for the future yet...
M.
your offspring should be congratulated
Well, given that we live within the Aneurin Bevan Health Board not too far away from that man's constituency, and that said 14 year-old spent quite a lot of time in hospital as a baby, he is acutely aware that he probably wouldn't have had any younger siblings if we had been paying off for all that treatment. Either that, or he wouldn't have been as fit as he is now.
M.
The Highways Agency have an IT location on a flood plain next to the M1 which is above them on an embankment. They know where they are and have built a moat around it and some very powerful pumps. In the event of a flood, the pumps engage and drain the moat by flinging the water OVER the M1 to the other side of the embankment.
I gather the test was spectacular.
The Highways Agency have an IT location on a flood plain
I heard a radio programme a couple of years ago on the new National Archives building in London, right next to the Thames. If I remember correctly, they put their IT in the basement because they would much rather the IT department (which presumably has decent off-site backups) flooded, than they ended up with a lot of soggy 1,000 year-old parchment.
M.
The Thames is pretty much a safe bet for building a DC next to because Thames barrier etc.
Still a waste of nice river real estate!
IBM have had their DC on the south bank for at least half a century and whilst it's pretty crap - I don't believe it's been wetted yet !
I can think of a certain basement area next to the Thames that's been flooded in the last 20 years. But that was down to a large water main running parallel to the river cracking open and the water finding its way through the ancient, long since covered and built upon, tributaries of the Old Father.
I don't know if it houses a DC or not; I suspect it does.
"In many shops you can only invoke the disaster process (if indeed you have one) with senior management approval."
The same goes for Academia, one University with a leading computer science dept finally moved it's below ground and often flooded data centre to an off-site location, has had a £175,000,000 fire due to an UPS problem (wheeling it in and simply plugging in a 300KVA UPS without checking the battery bolts is simply stupid) and has installed a total VOIP phone 'solution', in the event of power failure there are NO communications. Then during the recent bad weather the COO on a £250,000 salary followed Police advice stayed home as did the director of H&S, leaving it up to a former Household Cavalry officer and deputy VC, the VC was 'unavailable', to make the decision to close and send people home. He typical to type decided impassible roads, no buses etc was no reason to shut the site, they ended up bedding down stranded staff and students in the library!
Those of us that also followed Police advice, like the COO and H&S director, not to travel also lost a days pay or holiday as we can't tele-work like they claim to have done...
Reminds me of the time when a satellite office network failed.
Raise a help desk ticket? No, network is down.
Phone help desk? No, the phones were voice-over-IP.
Mobile phone help desk? Anyone know the real number? It's on the intran... Oh.
Someone had the page open, so rang it. To find external calls were barred.
Eventually solved via a personal call to a colleague's mobile at the main site who scurried down to the depths of the IT cave and caressed the offending router.
What with living in Earthquake Country (SF Bay Area) that's why we have a POTS line separate from the cable system broadband. If my spaceship ever comes in and I can move to where I'd like to live and in a style to which I'd like to become accustomed, I'd get a dual-WAN router and have broadband from *both* phone company and cable company.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
