Expired cert... Really? #O2down meltdown shows we should fear bungles and bugs more than hackers It's a bit of a cliche that "everything's connected", but O2's stunning outage yesterday – chalked up by Swedish kitmaker Ericsson to an expired software certificate – is a reminder of how true that is. Payment terminals croaked, bus displays went blank. Strangers blinked at each other in the street, like Robinson Crusoe …
FFS (For F£$k Sake) expand your acronyms the first time you use them!
I've got better things to do on a Friday mid-morning than work out whether M2M means made to measure, machine-to-machine, or some defunct Norwegian pop duo!
Well, slightly better, I mean - reading the Reg ......
MVP, what counts for any normal techy solution in the current day. Deliver the absolute minimum, promise the earth & walk away, safe in the knowledge that unless the customer is really, really big there is sod all anyone can do about it.
And even if you are really big, this is still probably sod all you can do about it.
Came up on the first Google search so it must be right.
Acronym = Letters that from words
Abbreviation = Shortened word E.G. St, Dr etc
Initialism = First letter of each word and enunciated E.G. VIP
If I'm wrong blame Google, it's not that I'm lazy... honest!
Do not blame Ericsson here.
UK telco operations have a well established and entrenched fear of certificates for anything.
Once upon a time, before I went back to write software, I still did network architecture including security aspects. So while working in a major UK telco I proposed the idea of certificates everywhere for purposes of inventory, identification and security of provisioning. I was freshly out of a vendor where I did most of the design and implementation of a x509 retrofit into everything and they became the foundation of how the system fits together. So I was expecting some questions or a technical discussion.
I got none.
The faces around the table looked like they were a still frame from The Shining. They looked at the idea like I was serving a disemboweled body with maggots and suggesting they eat it. They were horrified at the idea despite having less than 60% accurate inventory and a long standing requirement to secure key aspects of the network management.
This fear has its roots in incidents like the one in O2. It is also the root cause of incidents like O2.
UK telcos (and most telcos in general) fail to understand the most basic principle of using X509 for infrastructure purposes.
It is: YOU RUN YOUR OWN CA. No vendor roots. The root is yours. And so are ALL certs.
Because they do not understand it and fear it, they either use vendor certs (which expire at the most unfortunate moment) or outsource it to an external CA which defeats the purpose of the exercise as you are no longer in control of your network. Either one of these results in an incident like O2 which in turn results in more fear, more vendor use and more outsourcing.
Ad naseum, rinse repeat.
Oh, and by the way, no lessons will be learned from this incident - O2 will NOT start running its own CA as it should.
In what electronic diary? Notifying whom?
Do you know how many certificates large enterprises have to manage now? It would be a full time job for someone - but if you made it that, you'd be screwed when they went on vacation or quit and the reminder from their electronic diary went to /dev/null.
The whole system around certificates is irretrievably broken if you require humans to be in the middle of it. It has to be automated - a subscription service that automatically updates. We will never see the end of such issues so long as humans have to be "reminded", because we are fallible. If the certificate for some weird page hardly anyone visits expires, it might be weeks before the company is notified. If the certificate required for mobile data to work at a large provider expires, it could do a lot of damage in the hours required for the problem to be diagnosed and corrected.
The whole system around certificates is irretrievably broken if you require humans to be in the middle of it. It has to be automated - a subscription service that automatically updates.
Suggest you dust down the risk assessments from the mid-1990's for Single-Sign-On solutions - these worked well whilst everything worked, break something and everything fell into a rather big heap, from which it was easier to reset and start again than trying to recover...
The obvious issue with subscription services is ensuring the bank account(s) from which monies are automatically taken always have sufficient funds (or haven't been closed) and if there is a hiccup in payment processing things get escalated so that action can be taken before certificates expire...
True, payment processing can be a problem, but no more of a problem than it is for manual payment. Ideally it would be done with a yearly subscription for all your certificates in a lump sum, or paid in monthly installments, rather than dribbling out a small payment each time a certificate is renewed. The accounting department would HATE YOU if you managed 3000 certificates and each was a separate charge for yearly renewal!
Automated renewal also makes it practical to have certificates that last only a month, making the cumbersome process of revoking them if compromised less of a factor.
If your organisation relied on certificates and you were using more than a handful, I suggest you would be well advised to set up your own PKI, it isn't all that difficult. That would reduce your 3000 certificate (subscriptions) to one root certificate.
It also makes it practical to have as you suggest short lived certificates as they would be wholly managed within your own infrastructure.
BTY, if your Accounts department can't handle 3000 certificate renewals a year then there is something wrong with it - its not that difficult in many accounts/financial systems to set up a bank account and ledger for reoccurring IT expenditure/subscriptions. But I expect the problem is that in many companies IT doesn't talk finance to Finance and so get things neatly structured.
And that still requires a manual process to insure EVERY certificate finds its way into that electronic monitoring system. This is better than a manual process around every renewal since you only need to do it once for a certificate and then you are good for as long as that particular certificate-requiring function remains exactly the same.
Better, but not good enough.
Cheap almost free open source monitoring software can keep an eye on certificates and give you prior warning that the date in one is approaching. You can choose how much warning you want and it will display it on a dashboard in red, ,send you an email or automatically open an ITIL compliant helpdesk ticket for you, with P1 urgency if you want.
Even the most shoddy IT shops I've dealt with have this sorted. It's really simple stuff.
werdsmith, your missing a vital point, your assuming O2 (the company) actually give a fook (shareholders will if share price slides longer than 24hours).
Give it a week and nobody will even remember they had an outage, once they can upload fish face pictures to instatwat or pictures of their lunch to twatbook
I can see what you're getting at. The certificate system has a different purpose for this situation. It isn't about somebody such as me, downloading software from a myriad of possible suppliers, possibly via intermediaries, where the certificate is about blocking access to possible malware, now with such things as HTTPS. Secure delivery still needs attention, but once a genuine copy of the software is delivered and authorised for use, the supplier's action (or inaction) shouldn't be able to stop it working.
Yeah, I suppose contracts can set up something like software rental, and that's nothing new. But if you shut down your customer I am sure the lawyers would be interested in the procedures you followed.
"But a look in their forums shows tons of people just screaming at them, who didn't even bother reading the news."
How were they supposed to read the news when their phone data connection was down? You don't honestly think they would have something old fashioned like a landline based connection or a radio or even a TV, do you? No, of course not. The world had just ended!
"How were they supposed to read the news when their phone data connection was down?"
How were they able to post in forums if they had no data connection...
I suggest that those able to access forums weren't those truely impacted by this outage, who's smartphone would have been reduced to a games console for Snake and Tetris (aside: showing my age here)
Giff, Gaff, you mean Telefonica aka O2?
Maybe its just me but their adverts really get on my goat, moreso than any other telcos ads (which are bad) every add they spout all i can hear in my head is Liar Liar Bums on fire, your telefonica in disguise you charlatan!
replace Giff Gaff with Tesco, Sky and Lyca......it fits!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
