back to article Why millions of Brits' mobile phones were knackered on Thursday: An expired Ericsson software certificate

Ericsson says an expired software certificate caused the outage that left tens of millions in the UK unable to call or text from their mobile phones, nor use 4G connections, on Thursday. The Swedish equipment maker, which manufactures much of the backend gear in the world's cellular networks, said today the downtime was due to …

  1. FuzzyWuzzys

    Don't feel so bad Ericsson, you probably did us all a favour!

    How may "zombies" had to stop staring at phones on public tranport and actually read something in the paper, look around them or worse actually talk to other passengers! Oh the humanity! Oh the number of cat videos and half-naked teenage girls who's Instagram pages didn't get visited today, oh think we need a charity single by Sting and Bono to help them through this terrible time.

    Yes, I know there probably was serious fallout for businesses and people urgently trying to arrange personal business but for most of us it's just bloody annoying and for the most part we had to take a break from our screens and actually take in the world around us for a day.

    1. Teiwaz

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      or worse actually talk to other passengers!

      Talk about expecting miracles.

      I remember using public transport more often than I do now, back before mobile phones or even smart phones.

      I don't recall random passengers striking up conversations with others on public transport then either*.

      But perhaps the technique of sitting frozen frigid in embarrassed silence and trying not to make eye contact with anyone is a lost skill now.

      * Well, maybe once, and I'm fairly sure she was a visitor from another dimension.

      1. Nick Kew

        Re: Don't feel so bad Ericsson, you probably did us all a favour!

        Damn, I must be a freak. On a long train journey, I more often than not find myself in conversation with one or more actual people, merely by virtue of occupying neighbouring seats.

        p.s. my O2 4G returned sometime yesterday evening. When I put the phone on the charger around midnight, it was there.

      2. jmch Silver badge
        Happy

        Re: Don't feel so bad Ericsson, you probably did us all a favour!

        "I don't recall random passengers striking up conversations with others on public transport then either"

        Me neither. Before phones there were books and newspapers. And the current craze for giant headphones instead of tiny earbuds is just a re-run of the late 80s / early 90s although of course back then they weren't noise-cancelling and actually broadcast the sound to the outside world as much as to the listener.

    2. tin 2

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      Don't be daft, I just tethered to my other phone and carried on being a zombie.

    3. Gene Cash Silver badge

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      > How may "zombies" had to stop staring at phones on public tranport and actually read something in the paper, look around them or worse actually talk to other passengers

      Not me! I have Solitaire on my phone for emergencies such as this!

    4. Jamie Jones Silver badge

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      I was on the tube one evening, about 8pm. It wasn't packed, but most seats were taken.

      Everyone was doing the usual - reading the adverts, looking at their phones, trying to avoid eye-contact.

      Then at one stop, 3 or 4 people got on.. Shall we say "in the party spirit"... They were singing, and talking to the rest of us, and cracking jokes with us, and goading us all into generally joining in.

      The whole carriage joined in, and started cracking jokes too. Even when these people got off, everyone else on the carriage continued chatting, and everyone said "bye" when they got off at their stop.

      Just needs an ice-breaker...

    5. Anonymous Coward
      Anonymous Coward

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      “look around them or worse actually talk to other passengers!”

      What sort of monster are you?

    6. Jeffrey Nonken

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      OK, grandpa, you've identified a problem: things are changing in ways that you don't like. You've expressed dismay and contempt, which thousands of others have done before you.

      Do you have a solution to propose? I mean, other than simply shutting down the cellular network and/or the Internet, which (aside from the problems doing so and making it stick) has its own negative consequences.

      Because if not, you're in danger of looking like Abe Simpson. https://i.kym-cdn.com/photos/images/newsfeed/001/044/247/297.png

    7. Anonymous Coward
      Anonymous Coward

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      You realise that “newspapers” (and magazines, like the one we are all reading here) are on the internet now?

      The idea of printed newspapers nowadays is terribly retro, so why would I have a bulky inky smelly non-virtual one…?

      Rightly or wrongly pretty much the only people who buy them are other journalists and similar PR or media people.

    8. Teawain

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      Imagine the shortage of digital dopamine from not being able to share, like or post anything. Tragedy lol.

  2. TheProf

    Reminder

    Google Calendar.

    December 5 2019

    Renew certificate.

    1. Anonymous Coward
      Anonymous Coward

      Re: Reminder

      And then you leave, and no-one else has access to your calendar/mailbox.

      No problem, you say, use a group calendar! And then due to reorgs/scope creep/laziness, "your" groups calendar falls into disuse. Or the mailing list gets retired. Or the recipients filter annoying certificate-provider emails to trash.

      Yes, I've seen them all. Though the "best" was a replacement root certificate, replacing a perfectly good root cert, but not published to the thousands of systems that depended on the Certification Authority chain.

      Certificates - Great in theory, and they even tell you exactly what's gone wrong, but will bite the unwary.

      1. Teawain

        Re: Reminder

        ... reminds me of >

        https://www.theregister.co.uk/2003/11/06/microsoft_forgets_to_renew_hotmail/

      2. Anonymous Coward
        Anonymous Coward

        "but will bite the unwary"

        I'm surprised about how many applications using certificate, don't have any kind of management and warnings about them. You have to manage everything "out-of-band", and even most CA software more or less think they're done as soon as they issue a certificate, and doesn't make management and especially warning very friendly.

        Often, applications certificate features looks "bolt-on" somehow, and nothing is done to tell when a certificate is about to expire. All the telemetry, tracking, big data analysis an nothing warns when a damned certificate is about to "die"???

      3. jmch Silver badge

        Re: Reminder

        "No problem, you say, use a group calendar! And then due to reorgs/scope creep/laziness, "your" groups calendar falls into disuse. Or the mailing list gets retired. Or the recipients filter annoying certificate-provider emails to trash."

        Yep. I was thinking - how about the issuing certificate authority, which knows all the certificates issued, to whom, and when they expire, sends a notification to certificate holders whose certificates are about to expire, but same problem remains - who do they send it to?

        That's the problem with medium term certificate duration. If it's issued for 20-25 years it would be obsolete by the time it has to be changed (but too insecure). If it's valid for max 6 months or 1 year there would be enough attention on it to not forget about it, renewal would be something that gets done in the quarterly or annual business cycle (but too frequent might be a PITA). 2 or 3 years is the sweet spot for it to be forgotten about!!

    2. Anonymous Coward
      Anonymous Coward

      Re: Reminder

      perhaps it was somebody streamlined out, who "forgot" to mark a date in the calendar :D

    3. Anonymous Coward Silver badge
      Paris Hilton

      Re: Reminder

      And your successor thinks: OK, so I need to renew a certificate... but which one? Then proceeds to go off to renew their 50m swimming certificate.

      1. Aladdin Sane

        Re: Reminder

        That would be Arnold J Rimmer, BSC SSC.

    4. SImon Hobson Bronze badge

      Re: Reminder

      From the way it's written, this doesn't sound like the security certificates people here seem to be assuming. A lot of software like this uses keys (or certificates) to enable features - when it runs out, the software/feature stops working. Thus you have to keep paying the vendor's support fees for as long as you want to keep using the software/feature.

      And typically there is some management function that will a) warn you about impending expiry, and b) allow installation of new keys/certificates.

      It sounds a lot like "something went wrong" with this renewal process, so come the expiry time of the key, the software/feature stopped working - and the network stopped working.

  3. macjules

    Note to self ..

    Next time replace that Symantec certificate ...

    1. Morten Bjoernsvik

      Re: Note to self ..

      use LetsEncrypt and Certbot

      1. TimMaher Silver badge
        Pint

        Re: Note to self ..

        Bummer. I was going to write <code>sudo certbot-auto</code>.

        Have an up vote.

        1. choleric

          Re: Note to self ..

          That works great until your internet connection goes down, or the server gets firewalled by someone who doesn't understand certbot...

          1. HMcG

            Re: Note to self ..

            If you allow people who don't know what they are doing to have access to your server firewall rules, you have bigger problems than you yet know...

            1. choleric

              Re: Note to self ..

              Yep, that's exactly the point isn't it? Someone sets something up, assuming that the system will work as infinitum, but it ends up being forgotten by someone else in the system.

              It doesn't have to just be server firewall rules. It can be something upstream, eg. a new router, that quietly locks out regular but infrequent network activity. The server admin is not necessarily the network admin. No one notices until it's too late.

              The result is a popcorn moment.

        2. KitD

          Re: Note to self ..

          Even better, set up a certbot renew cron job

      2. tomalak

        Re: Note to self ..

        These things don't have internet access. They're not a hobbyist website. They're core nodes in a telecom network. It's national infrastructure.

        1. theblackhand

          Re: Note to self ..

          “These things don't have internet access. They're not a hobbyist website. They're core nodes in a telecom network. It's national infrastructure.”

          Yes....My question is if “older software” means that a fix was available via an existing patch or upgrade that had been “delayed” or whether this was a new and unexpected issue.

          I don’t expect that even with Internet access that the certificate could have been renewed automatically.

        2. Anonymous Coward
          Anonymous Coward

          "These things don't have internet access"

          Well there's the problem straight away - no wonder none of the traffic was able to access any data based web services.

          And wow it's going to be a slow process with an engineer visiting every box with their serial cable to update the certs.

          1. Anonymous Coward
            Anonymous Coward

            "And wow it's going to be a slow process with an engineer visiting every box with their serial cable to update the certs."

            Why do think it took so long to restore?

          2. Kevin Pollock

            Let's just clarify this.

            They don't have INTERNET access, but they are networked. They are connected to something called the DCN (data comms network). These days a DCN is an air-gapped IP network using private addressing. On most comms gear there is a designated DCN port. Inside the device the DCN port must be entirely (ie. no electrical circuit connection) separated from any internet traffic that might flow through the device. A Cisco or Juniper router that provides internet connection, for example, will also have a DCN port - but that port must be totally "air gapped" from the traffic ports in the device.

            The DCN is a separate IP network run by the service provider and it has no internet access at all - because that is one of the main things that prevents it from being hacked.

            It's particularly important that Management Systems do not have Internet connections - unless you want them nuked by a DDoS attack.

            You can still manage SW updates etc. centrally, and you don't need to send engineers out with memory sticks and craft terminals unless something goes horribly wrong.

        3. PerlyKing
          WTF?

          Re: These things don't have internet access

          @tomalak Call me a dillettante dabbler, but are you telling us that the "core nodes in a telecom network" which provides Internet access to millions ... don't have internet access?! Me not understand X-(

          1. Doctor Syntax Silver badge

            Re: These things don't have internet access

            but are you telling us that the "core nodes in a telecom network" which provides Internet access to millions ... don't have internet access?

            No longer having internet access was the problem.

          2. Anonymous Coward
            Anonymous Coward

            "which provides Internet access to millions ... don't have internet access?"

            I really hope so. I hope they are reachable for management only from an internal management network separated from the Internet traffic they carry. I really do no expect any management access being connected directly to the Internet.

            These devices are used by the very companies that build the core network infrastructure, they should not need "the internet" or any other network to be reached by the control rooms...

            Still, if the certificate was used for the management network access....

    2. Anonymous Coward
      Anonymous Coward

      Re: Note to self ..

      tbh it was probably a certificate that had a reeeeally long expiry date. Maybe 10-plus years. Hence why it took so long to sort out?

  4. A Non e-mouse Silver badge

    More detail

    Was this software administered by O2 or Ericsson? 'Cause one of them needs a huge slap for missing that deadline.

    1. JetSetJim
      FAIL

      Re: More detail

      More to the point is why the fsck the s/w doesn't present a big flashing dialog stating "Certificate about to expire for <SOFTWARE_COMPONENT>, please renew or lose all packet data connectivity for your subscribers on <EXPIRY DATE>" every time anyone logs in to the management s/w when such a scenario becomes likely (e.g. for the last month). This should be a basic part of any s/w licensing feature.

      1. RegGuy1 Silver badge
        Facepalm

        why the fsck the s/w doesn't present a big flashing dialog

        Opps! Sorry that was me. I must have kicked the reminder machine that is under my desk, and I think I dislodged the network cable.

      2. Jellied Eel Silver badge

        Re: More detail

        More to the point is why the fsck the s/w doesn't present a big flashing dialog stating "Certificate about to expire for..

        .. failing to insert money. Back in the simpler days, you bought kit, it had software, you paid for a service contract that supported it, including access to software updates. Then along came software as a service, and new revenue streams. So instead of buying kit, you pay an annual rent or it can stop working.

        Which can (or should) factor into vendor selection given it can work out to a lot of money, especially if vendors want $$$ every time you add a device, or in some cases just add a new virtual circuit. Alcatel's NMS used to work on that model where you bought licence packs of points, and actions cost points. They weren't selected for a large network I worked on mainly for that reason. Nice kit, lousy business model.

        I'm a bit suprised that this happened. An expired cert should have been flagged as a critical risk, if that resulted in a network shutdown. Plus given $$$ for new licences, a sales bod should have been chasing for renewals.

        1. S4qFBxkFFg

          Re: More detail

          "I'm a bit suprised that this happened. An expired cert should have been flagged as a critical risk, if that resulted in a network shutdown."

          It should be considered almost as important as filing the annual accounts - what do the tax authorities respond with if someone forgets that?

      3. roblightbody

        Re: More detail

        As long as someone actually logs into the management console... or is looking at the notitication alert emails that its sending out.... thats if someone has actually configured it to send out emails....

        1. Doctor Syntax Silver badge

          Re: More detail

          "thats if someone has actually configured it to send out emails...."

          And if the recipient of the emails is still there.

          It's easy enough to set up a warning system. Protecting that warning system against the ravages of management changes is a different matter and almost certainly outside the powers of whoever set it up. If you were the one who was the designated recipient of the email and you've just been booted out of the job are you going to be in a mood to warn whoever did the booting that that particular mail box needs to be monitored? Is the booter even going to listen if you did? And will the booter get booted out in the next bout of changes?

          There needs to be personal responsibility on those making such changes to ensure that everything like this gets covered under the new organisation. HMG has woken up to the fact that national infrastructure needs to be protected even when it's in private hands. Maybe that protection should extend to personal sanctions on those involved, even up to CEOs and board members. Make them sweat a little. After a few big personal fines or gaol sentences businesses would become a little less cavalier about reorganisations and outsourcing.

          1. Anonymous Coward
            Anonymous Coward

            "And if the recipient of the emails is still there."

            I hope nobody really uses emails for that anymore - but for small networks. What are SNMP and all those expensive network monitoring systems for?? Big red lights should appear besides any device which have certificates about to expire. It's akin having and hardware components about to fail. You get proactive SMART alerts, but nothing about certificates...

      4. JohnG

        Re: More detail

        "More to the point is why the fsck the s/w doesn't present a big flashing dialog stating "Certificate about to expire for ...."

        Perhaps this was the responsibility of people amongst the 18,000 laid off by Ericsson in the last year.

    2. PerlyKing
      Joke

      Re: One of them needs a huge slap

      Even better, a "limited number" of slaps. Or maybe that's just the PR department ;-)

    3. Anonymous Coward
      Anonymous Coward

      Re: More detail

      was thinking the same, is it Ericsson who install and manage it or O2, you'd kind of think O2 would have noticed via SCCM or some other monitoring tool that it is going to expire

  5. Anonymous Coward
    Anonymous Coward

    Hey ...

    My old-skool Nokia 3310 was working just fine.

    O2? Nah, not me! lolol

    1. MrMerrymaker

      Re: Hey ...

      Who cares then? You're not relevant!

    2. JimboSmith Silver badge

      Re: Hey ...

      Couple on the table next to me at lunch didn't have any service on their phones. Not to be outdone by Sky Mobile not working they switched to their O2 backup sims. Sadly that wasn't working for them either and therefore "Every network must be down at the moment not just Sky!" They were most amazed when I received a call.........

    3. Dave Lawton
      Happy

      Re: Hey ...

      So was mine, and yes, it's O2 (Tesco),

  6. Ryan 7

    Test successful

    At least we know the certificate is respected downstream!

    1. Anonymous Coward
      Anonymous Coward

      Re: Test successful

      The system is robust. The people, not so much.

  7. Anonymous Coward
    Anonymous Coward

    Boo hoo

    Millions addicts were unable to download interracial cream pies off ChatSnapApp.

    #firstworldproblems

    1. Anonymous Coward
      Anonymous Coward

      Re: Boo hoo

      And some people were unable to contact emergency services ( https://www.bbc.co.uk/news/business-46465839 ). #dontbeadick

      1. JetSetJim

        Re: Boo hoo

        I think you've mis-understood the article:

        The couple were eventually able to make contact via wi-fi and the emergency 999 number.

        Reads like "she collapsed, and tried to ring partner. Call failed, so dialled 999 successfully - presumably this either got routed via another network, or bypassed the bit of s/w that was down. If the latter, then this implies that the s/w had something to do with user authentication, which is not a part of an emergency call setup. Then perhaps Skyped her partner successfully.

        All networks *have* to service received emergency calls without doing any user/equipment authentication - indeed phones without SIMs in can make such calls.

        Still a pretty shitty thing to have happened.

        1. Anonymous Coward
          Anonymous Coward

          Re: Boo hoo

          You cannot call 999 in the UK without a SIM card

          1. Anonymous Coward
            Anonymous Coward

            Re: Boo hoo

            You cannot call 999 in the UK without a SIM card

            How confident are you of that? AIUI calling the designated emergency service numbers is the ONLY number you can call without a SIM card.

            1. Chuunen Baka

              Re: Boo hoo

              The SIM card's main job is authenticating the phone on the network. Calling emergency numbers does not require authentication.

            2. smalldot

              Re: Boo hoo

              Your understanding is correct as far as the original ETSI/3GPP standards definition goes. But there are countries that have national legislation saying emergency calls will not be allowed unless the phone has a SIM card (and transmits the identifier of said card to network). A quick Google search says UK is a country where a valid active SIM is required.

              I understand the need to reduce hoax calls by trying to identify idiots who makes them. But this can make a costly lesson for people from other countries who are used to having an extra phone laying around just for emergencies.

  8. Commswonk

    Thank goodness I'm retired...

    Prior to retirement I often had to investigate why "something" wasn't working properly. Now I would have to report that A complete and comprehensive root cause analysis is still in progress.

    While, I must assume, keeping a straight face throughout.

    1. tomalak

      Re: Thank goodness I'm retired...

      Yes it's annoying having to use words isn't it

    2. Anonymous Coward
      Anonymous Coward

      Re: Thank goodness I'm retired...

      Yes the buzzwords abound in any crisis from a large company.

      However I have witnessed many times that when the engineer looks for and finds why something is wrong they might be tempted to power cycle that piece of equipment and it magically works again. But then it keeps happening and so it keeps being power cycle until engineer 2 has a look and decides the best course of action is a reboot script that cycles the power every night at 2am when it's least busy.

      At least with a root cause analysis they would have spotted that a rogue GPO policy had inserted a scheduled task that consumed all the resources and didn't shut down correctly, or that the mythical cleaner was unplugging the gateway that the server depended on to do some vacuuming at midnight.

      1. Anonymous Coward
        Anonymous Coward

        Re: Thank goodness I'm retired...

        It's not the buzzwords, it's the long drawn out proof.

        Twice in the last few months I've had to do a RCA for some less than truly critical issues.

        I had the initial overview answer in less than half an hour i.e. 'It is very probably this', but no, a RCA must be provided.

        Waste around a couple of hours on each instance documenting the precise date, time, and sequence of operations that leads to :

        The issue being precisely what I'd said in the first case.

        Days of effort wasted over the course of a year, when instead things could be actually achieved, or I could be reading elReg.

        1. Killfalcon Silver badge

          Re: Thank goodness I'm retired...

          I have seen the "long drawn out proof" come up with a different answer to the original issue, but more often it's "As we suspected, X happened. This was because A, B and C did not stop X happening. We should retrain B, re-code A and correct the config in C so it's monitoring Production Y and not Dev".

          Or "the warnings have been going to an unrelated team in Birmingham for the last two years who, after three months of trying to get someone, anyone, to explain the warnings, set a rule to delete them on receipt."

          1. Anonymous Coward
            Anonymous Coward

            Re: Thank goodness I'm retired...

            The two most recent RCAs were :

            Yes, I know it works for this and not for that, that's because 'this' is at a different place to 'that' and you've moved the person doing it to a place where they can't do it any more. Might possibly prevent them asking again, if I'm lucky.

            and

            e-mail to the effect of 'you fucked this up, didn't you?' (when it wasn't even my responsibility to do this)

            response : pretty certain you ran this thing before you should have

            e-mail : 'no, we must have the full RCA as what you think is our fuck up is being questioned by someone else, and we want to shovel the shit on to you.'

            ..one long timeline later..

            You fucked up and ran it before you should have done. Have a nice day.

      2. Alistair
        Windows

        Re: Thank goodness I'm retired...

        @AC

        "Yes the buzzwords abound in any crisis from a large company.

        However I have witnessed many times that when the engineer looks for and finds why something is wrong they might be tempted to power cycle ........"

        I've never in my 30 years in IT seen a gateway do vacuuming for anyone let alone a server.....

  9. Anonymous Cowtard

    Days like today are when I'm thankful

    for my £120/year backup phone on a different network.

    364 days a year it's a slight annoyance, today it saved me thousands.

    1. katrinab Silver badge

      Re: Days like today are when I'm thankful

      My backup phone is a £10 top-up every 6 months or so.

      1. Anonymous Coward
        Anonymous Coward

        Re: Days like today are when I'm thankful

        My _main_ phone is a £10 top-up every six months or so.

    2. tin 2

      Re: Days like today are when I'm thankful

      I'm thankful my stupid company made me carry two phones.

      364 days a year it's a slight annoyance, today it saved my arse.

      1. Agincourt and Crecy!

        Re: Days like today are when I'm thankful

        "I'm thankful my stupid company made me carry two phones.

        364 days a year it's a slight annoyance, today it saved my arse."

        Weren't we told that people with 2 phones were either drug dealers or terrorists a while ago? Beware the big rubber glove!

    3. Anonymous Coward
      Anonymous Coward

      Re: Days like today are when I'm thankful

      What job do you work in where you can stand to lose so much without data for 1 day?

      1. Anonymous Coward
        Anonymous Coward

        Re: Days like today are when I'm thankful

        “What job do you work in where you can stand to lose so much without data for 1 day?”

        You know all those people that work out of the office that do everything electronically these days? Couriers, health careers that visit people at homes, maintenance staff that do home visits, sales reps, small businesses using card services. Pretty much anyone that used to get a laptop for their job and sync on Monday’s and Fridays when they were in the office. The list goes on....

        Is this a problem? I guess it depends how significant the effect was.

      2. JoshOvki

        Re: Days like today are when I'm thankful

        > What job do you work in where you can stand to lose so much without data for 1 day?

        Certificate management for Ericsson?

      3. Anonymous Cowtard

        Re: Days like today are when I'm thankful

        "What job do you work in where you can stand to lose so much without data for 1 day?"

        Data not a problem, it was phone calls on the road & on the job.

      4. katrinab Silver badge

        Re: Days like today are when I'm thankful

        "What job do you work in where you can stand to lose so much without data for 1 day?"

        Basically any job that involves visiting people to do things

        Courier / delivery

        Maintenance - where you visit customers to fix their things

        Sales

        Doctors

        Ambulance drivers

        Buses (my local bus company doens't use O2, but if they did, their ticket machines wouldn't be able to do card payments)

        Taxis / mini cabs

        Lots of others I haven't thought of.

      5. Anonymous Coward
        Anonymous Coward

        Re: Days like today are when I'm thankful

        Lots of multi stop driving jobs dependent on connectivity in dynamic scenarios where jobs get dynamically reallocated at head office so a driver will find their "job list" liable to change frequently, jobs often allocated via software on a tablet / phone via custom app as far more efficient than head office making voice calls / texts to staff telling them of changes.

        There's hidden stuff such as vehicle trackers, oft used in logistics, these have SIM card and send position updates, who knows how many of these used an affected SIM card? A lot of back end logistics is a PITA when you do not have accurate details on vehicle positions as dynamic reassignment needs location info.

        So all sorts of "road warriors" could have been affected.

        I know some companies that luckily avoided a big hit as their mobile apps and /or trackers ran on a different network, otherwise they would have been really struggling

  10. Hubert Thrunge Jr.
    Trollface

    Compensation required

    I demand compensation.

    I was unable to post a picture of my breakfast on instagram/twitter/facebook/etc... this morning.

    I also wish to know who to send the overtime bill to, as a shitstorm of emails hit me the moment I walked through the door at home and the WiFi connected.

    Thank you for not listening. It's been wonderful.

    1. FlamingDeath Silver badge

      Re: Compensation required

      The difference in opinions is really contrast when you compare BBC & TheReg Commentardery regarding this is exact subject, using the internetz for trivial none important things and then going into meltdown when it becomes unavailable

      If I remember correctly, The exact comment "I was unable to post a picture of my breakfast on instagram/twitter/facebook/etc... this morning." on the BBC got heavily voted down

      Is it safe to say that the great unwashed frequent the BBC more than here?

  11. Oldfogey
    FAIL

    Standby?

    If mobile data is crucial to you, why do you not have a plan for maintaining it in the event of problems?

    Have a spare mob on PAYG? or

    Have a phone not locked to a service, and a spare SIM in your wallet?

    A dual SIM phone? (Best option)

    If it is REALLY essential, and you have not taken precautions, I find myself very short on sympathy.

    1. Richard 12 Silver badge

      It was voice too

      Cellular network communication is the only option for a lot of people, including people who may have needed to call the emergency services yesterday, but could not.

      And if the ESN was live already, that would include the emergency services themselves.

      It's likely that several people were physically harmed by the voice outage. Any deaths may still be undiscovered.

      1. Anonymous Coward
        Anonymous Coward

        Re: It was voice too

        Airwave is a single network. Yes it does go down and has done a few times, sometime for significant periods albeit generally locally. The backup? A mobile phone or point-to-point.

    2. Zebad

      Re: Standby?

      Exactly. Relying on a single network = relying on a single point of failure, a basic configuration to avoid if you're setting up a resilient service.

      Anyway, my O2 4G is working again this morning, back to the cat pics.

    3. Doctor Syntax Silver badge

      Re: Standby?

      "If it is REALLY essential, and you have not taken precautions, I find myself very short on sympathy."

      Remember the people at the sharp end are not usually those taking the decisions. The OP of this particular thread was someone whose employers had done so. There will be others whose employers hadn't and even a few where an original dual provision had been cut to save money.

    4. irrelevant

      Re: Standby?

      My home broadband was down yesterday, too. (second exchange fault within a week, thanks, Sky.) But I'm prepared for that. I've a cellular dongle plugged into the firewall that is set up as fallback. Runs on O2...

  12. Fruit and Nutcase Silver badge
    Joke

    NHS

    Perhaps they should not be in such a rush to get rid of all those FAX machines

    1. Anonymous Coward
      Anonymous Coward

      Re: NHS

      Because someone needs to vote for PewDiePie.

    2. Doctor Syntax Silver badge

      Re: NHS

      Why the joke alert icon?

  13. error 13

    I thought all certificates were good until 2037..?

  14. MrMerrymaker

    Upsides abound!

    Had to get to a new site today and usually rely on Google maps or Moovit. (GPS works without a connection obviously but I did like the challenge)

    I found it by myself!

    And my phone was in my jacket all day. I've lost only a few percent.

    It basically was a forced airplane mode. And best of all...

    NO SCAM CALLERS

  15. Fruit and Nutcase Silver badge
    Facepalm

    OOO

    O2 Outage Outrage

    1. Anonymous Coward
      Anonymous Coward

      Re: OOO

      O2 Outage Outrage

      So that's O3, then? Or would it be O4?

  16. Terry 6 Silver badge

    I don't get it

    Surely,for organisations as big as that, with service as wide as that, they could find a way of bypassing the need for certificates that expire. Next thing they'll be blaming a lack of 50p pieces for the meter.

    1. Anonymous Coward
      Anonymous Coward

      Re: I don't get it

      My guess is the certificate is used for securing API calls between services within a larger system so end users don’t get to see the issue or get presented with a warning they can bypass.

      Given the time required to troubleshoot and resolve the issue, it will be interesting to see if we ever see the final report into the outage. The blame might lay with decisions within O2 (ie using software beyond its end of life or without critical patches) rather than with an external vendor. Pure speculation...

    2. Anonymous Coward
      Anonymous Coward

      Re: I don't get it

      Next thing they'll be blaming a lack of 50p pieces for the meter.

      There's a thought. Who knows what certificatey badness lurks on the inside of smart meters? Given that these have cranky software written by hardware makers, and there's zip all standardisation, we can probably expect some vast screw up in future years. And due to the idiotic specification created by government, when a smart meter has problems it defaults to cutting the customer off.

      At least nobody will be using energy without paying when the software gremlin visits.

    3. Doctor Syntax Silver badge

      Re: I don't get it

      "Surely,... they could find a way of bypassing the need for certificates that expire."

      They could also make things more convenient by not putting locks on doors etc.

      1. Terry 6 Silver badge

        Re: I don't get it

        Er no. Having alternative locks isn't the same thing as having no locks.

  17. Anonymous Coward
    Anonymous Coward

    Simple monitoring...

    The number of times I've set up monitoring for 'things that expire'.

    For a variety of 'things' and a variety of customers, from ESN to businesses to my own personal usage.

    1. Doctor Syntax Silver badge

      Re: Simple monitoring...

      "For a variety of 'things' and a variety of customers, from ESN to businesses to my own personal usage."

      Who or what receives the alerts from the monitoring? For your personal stuff, presumably you. But if those from your customers are handled by the customers themselves do you know if there's still anyone looking out for them? If they all come through to you then you become a single point of failure for the customers and what happens then if you retire or fall under a bus?

      It's not the setting up of something to raise alerts that's the problem. Monitoring is an on-going process and these days on-going processes are apt to be interrupted by system failures reorganisation by management, especially those processes that deal with rare events on demand.

      1. Killfalcon Silver badge

        Re: Simple monitoring...

        I wonder if there's a way to hook into org-charts, with notifications failing up the chain of command until they find a role holder who's actually in the system.

        Then you just produce reports that go somewhere higher up when a job role vanishes and isn't reassigned, and make sure HR know how to maintain it. They don't need to know what the report even means- just where it used to go. HR generally has sight of re-orgs, so they'll have very good odds of being able to work out who the new owner should be.

    2. Jellied Eel Silver badge

      Re: Simple monitoring...

      The number of times I've set up monitoring for 'things that expire'.

      Funeral director?

  18. tin 2

    An expired certificate....

    ...made the whole data network shut down?

    Wow. There was no point building in any other resilience elsewhere then.

    also, think the comment from Ericsson subtext is "o2 is still using very old software they should have replaced moons ago"?

    1. Doctor Syntax Silver badge

      Re: An expired certificate....

      "Wow. There was no point building in any other resilience elsewhere then."

      Do you mean a back door? Not necessarily a good thing when the whole point of the certificates is to secure the system.

  19. tomalak

    This is a really shocking faliure.

    However.

    If your bus timetable / iPad / etc is really that important, why do you have no redundancy? No backup? You're relying on one carrier, one network? That's pretty silly.

    Bet nobody will learn from it either. Just blame somebody else (O2) and wait for the next outage.

  20. Borkian

    Limited number of customers

    Considering how many people this hit I would really hate to see their definition of a large scale outage.

    1. John Brown (no body) Silver badge

      Re: Limited number of customers

      Yes, this.

      "network disturbances for a limited number of customers in multiple countries"

      "some 32 million people were without cellular service,"

      I suppose, technically, it was a "limited number of customers". It was limited to their entire userbase. "Limited" is one of those wonderfully ambiguous words which can imply "only a few" when used in the right way but is still technically correct when the PR is called out and it's actually everybody who is affected.

    2. Anonymous Coward Silver badge
      Boffin

      Re: Limited number of customers

      The customers were Ericsson customers - ie the telcos - not the end users.

      In the UK it affected O2, but not the other carriers (EE, voda), so yes "limited number" is appropriate.

  21. Dave Bell

    But what was working?

    I can confirm that SMS was failing, badly, over the O2 Network. I didn't sent many texts, but all were logged here as failed. One was delivered 20 times. Nothing critical for me, but annoying.

    I didn't try voice. Was that they only thing working?

    1. Slef

      Re: But what was working?

      Yep Voice was working, pity you did not try using your phone to make a ..........telephone call!! ;-)

      1. ItsMeDammit

        Not for me, it wasn't. About 15:00 my phone briefly dropped off the network entirely and into emergency calls only. It remained like this during my drive home (so many cell site hand-overs) and finally got back up onto it's home network about 18:30.

        As of 06:10 on 07/12 and statements to the contrary, I still problems sending text messages and have no data connection in Andover. The usual forced re-registering advice of airplane mode or turning the phone off and on again makes no difference. For me, O2 is still stuffed.

    2. jaygeejay

      Re: But what was working?

      Some calls, some of the time, didn't get through.

    3. Richard 12 Silver badge

      Re: But what was working?

      Voice was down for many users for 10 to 12 hours, as 4G carries voice as well as data.

      It was worse than the cells being gone, as handsets would register with O2, but be unable to make calls. So even "emergency only" mode wasn't available.

      Credit to them, O2 clearly prioritised getting voice back up. The "fix" seems to have been to shut down 4G network-wide thus forcing all phones to switch to the older network.

      1. Down not across

        Re: But what was working?

        Voice was down for many users for 10 to 12 hours, as 4G carries voice as well as data.

        Yes it does, but is should fallback to 3G/2G.

        Sometimes phones/cells dont fallback to older tech as they should. In that situation, it may be worth forcing the phone to 3G/2G in settings. I had an issue years ago where calls were not getting through and O2 (yeah, funny that) suggested forcing the handset to 3G which worked around the issue. Allegedly there was some issue (which persisted for some time) in the local cell(s).

    4. katrinab Silver badge

      Re: But what was working?

      I had "no service" all day, so my phone was working as effectively as my iPod.

  22. Frumious Bandersnatch

    I see that...

    Softbank in Japan is also experiencing problems, and since it's carrier for many other telcos, lots of similar problems are happening over there, too. Same reason?

    1. james 68

      Re: I see that...

      Y! Mobile also, and yes, it's because of the same dodgy Ericsson cert.

    2. Anonymous Coward
      Anonymous Coward

      Re: I see that...

      Yes, in Chile and Mexico too

  23. Anonymous Coward
    Anonymous Coward

    Certs as DoS

    I have several trust certicates disabled on my Android device to stop certain apps from communicating with their mothership.

  24. Anonymous Coward
    Anonymous Coward

    NHS no trust

    Just why was this trust relying on a mobile signal rather than WiFi connected to their own network? I can’t believe these iPads were all used outside on a random street.

    1. Anonymous Coward
      Anonymous Coward

      Re: NHS no trust

      The limited experience I have with health is that there is a significant (around 20%) of the staff that are typically working in either “mobile” surgeries such as immunisations or other temporary facilities or visiting patients in their homes or third parties such as rest and care homes.

      I’m guessing more this will vary based on the geographic area covered by the NHS trust and services offered.

  25. Kaltern

    No sympathy whatsoever.

    If your business is SO dependent on mobile networking, then, in the same way you always make a safe, reliable set of daily backups for your important data, you should already have a contingency plan for such events. Such as a cheap PAYG secondary driver.

    Blaming o2 for your 'loss of business' is like saying 'I always meant to get those backups done...'. And people are asking for compensation? Seriously?

    And the way the Sky news anchor was absolutely ripping into the CEO of o2 last night was ridiculous - literally asking if he was going to resign, and virtually demanding financial compensation...

    I would imagine there is plenty of contract clauses that cover unforseen issues such as these. I think the country as a whole needs to stop jumping on things like this - we're ALWAYS so quick to blame everyone else for our own oversights.

    (I'm not suggesting that there are not SOME legitimate complaints about what happened - but these will be special cases, and I imagine o2 will already be dealing with such companies due to contractual obligations)

    1. John Savard

      Re: No sympathy whatsoever.

      That sort of thing costs extra. And there's no good reason why computer systems shouldn't just work, forever, perfectly. Like the Sun keeps shining each morning. After all, the underlying physical circuitry doing the computations is highly reliable. All that's required is that the software be written correctly, once, the first time.

      Now, hard drives have moving parts, though, so one does have to allow for them failing. Trouble is that vendors have failed to offer an inexpensive, easy, and convenient method of backup. This is why desktop computers don't routinely come with tape drives for which the tapes have a capacity even close to a terabyte so you could do a complete backup to a handful of them.

  26. Anonymous Coward
    Anonymous Coward

    Alerts

    From my experience of network equipment there's usually alerts that pop up prior to certificates expiring.

    I don't know about transmission network stuff but it would be interesting to know if the Ericsson equipment did send alerts and the Operations team ignored it.

    Do see that a lot in places where they've outsourced Operations.....just saying...

  27. peterkay03

    Commercial....?

    I wonder if the real truth will ever come out; having been involved in commercial finance It would not shock me if this was simply a case of unpaid bills and Ericsson carrying out a threat to not renew the certs unless a very long outstanding bill was paid! The bigger the organisation the braver they get with not paying their bills just assuming the smaller suppliers won't dare withhold service....

  28. dcloud9

    Auto renew certs

    Use letsencrypt and forget cert expiration.

  29. defiler

    Remember that time when cellular data was faster than WiFi? And because of spectrum was always going to be better? And we were all supposed to ditch our WiFi and just use cellular data because it's awesome?

    Aye. That.

  30. Ben1892
    Black Helicopters

    Man in the middle

    Seems a convenient time/place to insert some GCHQ monitoring if you ask me ...do you want to trust this cert chain? Sure I do!!, I want to get the services up and running asap

  31. gungho

    I supposed setting a reminder for the certificate expiry wouldnt have worked ???

  32. Anonymous Coward
    Anonymous Coward

    Oops

    Expired cert forgotton about. We did that too once. Took out half the network. Certificates and their expiry dates are well monitored now.

    1. Herring`

      Re: Oops

      Certificates and their expiry dates are well monitored now.

      You're lucky. Place I worked at, every bloody year on this one system the cert would expire and people would flap around like headless chickens trying to find out why it had mysteriously stopped working. The security people in the US wouldn't let us use a cert with a longer expiry.

      1. stiine Silver badge

        Re: Oops

        Then you have to renew it every 11 months. Why is that so hard?

        Although I agree that it should be 375 days so you can effectively schedule it every 12 months.

        1. Herring`

          Re: Oops

          Then you have to renew it every 11 months. Why is that so hard?

          I know that, and you know that. But the people whose job it is to renew certs, not so much. Aided by the purchasing department that can't just go on the internet with a credit card but has to find out if it can be "bought more cheaply somewhere else".

          1. TonyHoyle

            Re: Oops

            I tend to find companies with that mindset are complete shitshows.. they waste more money trying to be cheap than they ever save.

            Then they go bust when all their cheap stuff breaks, and they're surprised.

  33. NXM Silver badge

    Bus timetables

    People would've had to look up from their screens to look down the street in case a bus was coming.

    My god, the horror.

  34. adam payne

    the supplier said, rather downplaying tens of millions of screwed-over punters as "a limited number of customers." Factually correct as O2 customers aren't their customers.

  35. the-it-slayer

    Phone calls... wassthat?

    Millennials discovered the use of phone calls and SMS for the first time whilst they were disconnected from their instagram counterparts.

    Productivity went up in universities and businesses across the country by 200%.

  36. wyatt

    Humm, the reg reader that commented about ESN needs to clarify things. ESN is going to be run on EE/BT so wouldn't have been effected by an O2 outage. ESN also has a separate system to EE's normal users so I'd 'expect' it to be isolated from an issue like this.

    Not to say that this won't effect EE some day, I'm sure there have been outages of Airwave at times.

    Also to add that 999 calls via a mobile device can be made via other providers networks. Has it been confirmed that 999 was unavailable to handsets with O2 Sims? I've made calls with an EE sim and no signal into the 999 network, I've no idea which provider routed it though.

  37. Anonymous Coward
    Anonymous Coward

    Everything was absolutely fine on EE. Which has built-in backup as it uses Orange and T-Mobile anyway.

    1. Killfalcon Silver badge

      With EE, I'm always a bit curious as to how disparate the two original company's kit still is. They've been EE a few years now, I expect they've managed to work in at least a few single points of failure by now!

  38. David Nash Silver badge

    Faulty Software?

    So if it was an expired certificate, what's all this about "The faulty software that has caused these issues is being decommissioned" ???

  39. Anonymous Coward
    Anonymous Coward

    ""The [National Health Service] trust I work for has lost connection of all the Apple iPads that are used for patient report forms," a Reg reader told us. "This is extremely worrying seeing that every emergency service will be using a 4G network for the entirety of their critical communications. This outage would've put lives at risk if ESN was live!""

    Uh.... Nope.

    ESN is going to be on EE, not O2.

    Plus, the iPads should be using wifi wherever possible, as opposed to 4G connections when being used for the purposes that are stated here, which frankly is overkill anyway when there are numerous other tablet solutions, more ruggedised than the iPad for this purpose. Plus, the mobile data terminals that the emergency services are currently using are not facilitiated by O2, but Vodafone.

    Either the reader is clueless or hysterically full of self importance of themselves or their NHS "organisation", which no doubt while always skint, is never ever short of money for iPads.

    Eitherway, they're talking sensationalist guff, as the NHS tends to.

  40. John Savard

    Conclusion

    Well, this shows that making the operation of software dependent on certificates that can expire is a bad idea, as it creates an additional point of failure.

    I suppose, though, that these software certificates are needed for some security purpose, as there is an opportunity to introduce unauthorized software to the systems in question.

  41. N2
    Unhappy

    But

    I could not upload or view cat videos on Thursday, for a whole day...

    Where do I file for compensation?

  42. Arachnoid

    So.........

    All these hacks from "Foreign sponsored states" are a complete waste of time on their part all they need to do is intercept all the certificate reminders

  43. Anonymous Coward
    Anonymous Coward

    the network trope

    Network admin replaced to save money by an army of cheap disposable replacements who only think and deal with what is in the contract.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like