back to article Why millions of Brits' mobile phones were knackered on Thursday: An expired Ericsson software certificate

Ericsson says an expired software certificate caused the outage that left tens of millions in the UK unable to call or text from their mobile phones, nor use 4G connections, on Thursday. The Swedish equipment maker, which manufactures much of the backend gear in the world's cellular networks, said today the downtime was due to …

Page:

  1. FuzzyWuzzys

    Don't feel so bad Ericsson, you probably did us all a favour!

    How may "zombies" had to stop staring at phones on public tranport and actually read something in the paper, look around them or worse actually talk to other passengers! Oh the humanity! Oh the number of cat videos and half-naked teenage girls who's Instagram pages didn't get visited today, oh think we need a charity single by Sting and Bono to help them through this terrible time.

    Yes, I know there probably was serious fallout for businesses and people urgently trying to arrange personal business but for most of us it's just bloody annoying and for the most part we had to take a break from our screens and actually take in the world around us for a day.

    1. Teiwaz

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      or worse actually talk to other passengers!

      Talk about expecting miracles.

      I remember using public transport more often than I do now, back before mobile phones or even smart phones.

      I don't recall random passengers striking up conversations with others on public transport then either*.

      But perhaps the technique of sitting frozen frigid in embarrassed silence and trying not to make eye contact with anyone is a lost skill now.

      * Well, maybe once, and I'm fairly sure she was a visitor from another dimension.

      1. Nick Kew

        Re: Don't feel so bad Ericsson, you probably did us all a favour!

        Damn, I must be a freak. On a long train journey, I more often than not find myself in conversation with one or more actual people, merely by virtue of occupying neighbouring seats.

        p.s. my O2 4G returned sometime yesterday evening. When I put the phone on the charger around midnight, it was there.

      2. jmch Silver badge
        Happy

        Re: Don't feel so bad Ericsson, you probably did us all a favour!

        "I don't recall random passengers striking up conversations with others on public transport then either"

        Me neither. Before phones there were books and newspapers. And the current craze for giant headphones instead of tiny earbuds is just a re-run of the late 80s / early 90s although of course back then they weren't noise-cancelling and actually broadcast the sound to the outside world as much as to the listener.

    2. tin 2

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      Don't be daft, I just tethered to my other phone and carried on being a zombie.

    3. Gene Cash Silver badge

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      > How may "zombies" had to stop staring at phones on public tranport and actually read something in the paper, look around them or worse actually talk to other passengers

      Not me! I have Solitaire on my phone for emergencies such as this!

    4. Jamie Jones Silver badge

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      I was on the tube one evening, about 8pm. It wasn't packed, but most seats were taken.

      Everyone was doing the usual - reading the adverts, looking at their phones, trying to avoid eye-contact.

      Then at one stop, 3 or 4 people got on.. Shall we say "in the party spirit"... They were singing, and talking to the rest of us, and cracking jokes with us, and goading us all into generally joining in.

      The whole carriage joined in, and started cracking jokes too. Even when these people got off, everyone else on the carriage continued chatting, and everyone said "bye" when they got off at their stop.

      Just needs an ice-breaker...

    5. Anonymous Coward
      Anonymous Coward

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      “look around them or worse actually talk to other passengers!”

      What sort of monster are you?

    6. Jeffrey Nonken

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      OK, grandpa, you've identified a problem: things are changing in ways that you don't like. You've expressed dismay and contempt, which thousands of others have done before you.

      Do you have a solution to propose? I mean, other than simply shutting down the cellular network and/or the Internet, which (aside from the problems doing so and making it stick) has its own negative consequences.

      Because if not, you're in danger of looking like Abe Simpson. https://i.kym-cdn.com/photos/images/newsfeed/001/044/247/297.png

    7. Anonymous Coward
      Anonymous Coward

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      You realise that “newspapers” (and magazines, like the one we are all reading here) are on the internet now?

      The idea of printed newspapers nowadays is terribly retro, so why would I have a bulky inky smelly non-virtual one…?

      Rightly or wrongly pretty much the only people who buy them are other journalists and similar PR or media people.

    8. Teawain

      Re: Don't feel so bad Ericsson, you probably did us all a favour!

      Imagine the shortage of digital dopamine from not being able to share, like or post anything. Tragedy lol.

  2. TheProf

    Reminder

    Google Calendar.

    December 5 2019

    Renew certificate.

    1. Anonymous Coward
      Anonymous Coward

      Re: Reminder

      And then you leave, and no-one else has access to your calendar/mailbox.

      No problem, you say, use a group calendar! And then due to reorgs/scope creep/laziness, "your" groups calendar falls into disuse. Or the mailing list gets retired. Or the recipients filter annoying certificate-provider emails to trash.

      Yes, I've seen them all. Though the "best" was a replacement root certificate, replacing a perfectly good root cert, but not published to the thousands of systems that depended on the Certification Authority chain.

      Certificates - Great in theory, and they even tell you exactly what's gone wrong, but will bite the unwary.

      1. Teawain

        Re: Reminder

        ... reminds me of >

        https://www.theregister.co.uk/2003/11/06/microsoft_forgets_to_renew_hotmail/

      2. Anonymous Coward
        Anonymous Coward

        "but will bite the unwary"

        I'm surprised about how many applications using certificate, don't have any kind of management and warnings about them. You have to manage everything "out-of-band", and even most CA software more or less think they're done as soon as they issue a certificate, and doesn't make management and especially warning very friendly.

        Often, applications certificate features looks "bolt-on" somehow, and nothing is done to tell when a certificate is about to expire. All the telemetry, tracking, big data analysis an nothing warns when a damned certificate is about to "die"???

      3. jmch Silver badge

        Re: Reminder

        "No problem, you say, use a group calendar! And then due to reorgs/scope creep/laziness, "your" groups calendar falls into disuse. Or the mailing list gets retired. Or the recipients filter annoying certificate-provider emails to trash."

        Yep. I was thinking - how about the issuing certificate authority, which knows all the certificates issued, to whom, and when they expire, sends a notification to certificate holders whose certificates are about to expire, but same problem remains - who do they send it to?

        That's the problem with medium term certificate duration. If it's issued for 20-25 years it would be obsolete by the time it has to be changed (but too insecure). If it's valid for max 6 months or 1 year there would be enough attention on it to not forget about it, renewal would be something that gets done in the quarterly or annual business cycle (but too frequent might be a PITA). 2 or 3 years is the sweet spot for it to be forgotten about!!

    2. Anonymous Coward
      Anonymous Coward

      Re: Reminder

      perhaps it was somebody streamlined out, who "forgot" to mark a date in the calendar :D

    3. Anonymous Coward Silver badge
      Paris Hilton

      Re: Reminder

      And your successor thinks: OK, so I need to renew a certificate... but which one? Then proceeds to go off to renew their 50m swimming certificate.

      1. Aladdin Sane

        Re: Reminder

        That would be Arnold J Rimmer, BSC SSC.

    4. SImon Hobson Bronze badge

      Re: Reminder

      From the way it's written, this doesn't sound like the security certificates people here seem to be assuming. A lot of software like this uses keys (or certificates) to enable features - when it runs out, the software/feature stops working. Thus you have to keep paying the vendor's support fees for as long as you want to keep using the software/feature.

      And typically there is some management function that will a) warn you about impending expiry, and b) allow installation of new keys/certificates.

      It sounds a lot like "something went wrong" with this renewal process, so come the expiry time of the key, the software/feature stopped working - and the network stopped working.

  3. macjules

    Note to self ..

    Next time replace that Symantec certificate ...

    1. Morten Bjoernsvik

      Re: Note to self ..

      use LetsEncrypt and Certbot

      1. TimMaher Silver badge
        Pint

        Re: Note to self ..

        Bummer. I was going to write <code>sudo certbot-auto</code>.

        Have an up vote.

        1. choleric

          Re: Note to self ..

          That works great until your internet connection goes down, or the server gets firewalled by someone who doesn't understand certbot...

          1. HMcG

            Re: Note to self ..

            If you allow people who don't know what they are doing to have access to your server firewall rules, you have bigger problems than you yet know...

            1. choleric

              Re: Note to self ..

              Yep, that's exactly the point isn't it? Someone sets something up, assuming that the system will work as infinitum, but it ends up being forgotten by someone else in the system.

              It doesn't have to just be server firewall rules. It can be something upstream, eg. a new router, that quietly locks out regular but infrequent network activity. The server admin is not necessarily the network admin. No one notices until it's too late.

              The result is a popcorn moment.

        2. KitD

          Re: Note to self ..

          Even better, set up a certbot renew cron job

      2. tomalak

        Re: Note to self ..

        These things don't have internet access. They're not a hobbyist website. They're core nodes in a telecom network. It's national infrastructure.

        1. theblackhand

          Re: Note to self ..

          “These things don't have internet access. They're not a hobbyist website. They're core nodes in a telecom network. It's national infrastructure.”

          Yes....My question is if “older software” means that a fix was available via an existing patch or upgrade that had been “delayed” or whether this was a new and unexpected issue.

          I don’t expect that even with Internet access that the certificate could have been renewed automatically.

        2. Anonymous Coward
          Anonymous Coward

          "These things don't have internet access"

          Well there's the problem straight away - no wonder none of the traffic was able to access any data based web services.

          And wow it's going to be a slow process with an engineer visiting every box with their serial cable to update the certs.

          1. Anonymous Coward
            Anonymous Coward

            "And wow it's going to be a slow process with an engineer visiting every box with their serial cable to update the certs."

            Why do think it took so long to restore?

          2. Kevin Pollock

            Let's just clarify this.

            They don't have INTERNET access, but they are networked. They are connected to something called the DCN (data comms network). These days a DCN is an air-gapped IP network using private addressing. On most comms gear there is a designated DCN port. Inside the device the DCN port must be entirely (ie. no electrical circuit connection) separated from any internet traffic that might flow through the device. A Cisco or Juniper router that provides internet connection, for example, will also have a DCN port - but that port must be totally "air gapped" from the traffic ports in the device.

            The DCN is a separate IP network run by the service provider and it has no internet access at all - because that is one of the main things that prevents it from being hacked.

            It's particularly important that Management Systems do not have Internet connections - unless you want them nuked by a DDoS attack.

            You can still manage SW updates etc. centrally, and you don't need to send engineers out with memory sticks and craft terminals unless something goes horribly wrong.

        3. PerlyKing
          WTF?

          Re: These things don't have internet access

          @tomalak Call me a dillettante dabbler, but are you telling us that the "core nodes in a telecom network" which provides Internet access to millions ... don't have internet access?! Me not understand X-(

          1. Doctor Syntax Silver badge

            Re: These things don't have internet access

            but are you telling us that the "core nodes in a telecom network" which provides Internet access to millions ... don't have internet access?

            No longer having internet access was the problem.

          2. Anonymous Coward
            Anonymous Coward

            "which provides Internet access to millions ... don't have internet access?"

            I really hope so. I hope they are reachable for management only from an internal management network separated from the Internet traffic they carry. I really do no expect any management access being connected directly to the Internet.

            These devices are used by the very companies that build the core network infrastructure, they should not need "the internet" or any other network to be reached by the control rooms...

            Still, if the certificate was used for the management network access....

    2. Anonymous Coward
      Anonymous Coward

      Re: Note to self ..

      tbh it was probably a certificate that had a reeeeally long expiry date. Maybe 10-plus years. Hence why it took so long to sort out?

  4. A Non e-mouse Silver badge

    More detail

    Was this software administered by O2 or Ericsson? 'Cause one of them needs a huge slap for missing that deadline.

    1. JetSetJim
      FAIL

      Re: More detail

      More to the point is why the fsck the s/w doesn't present a big flashing dialog stating "Certificate about to expire for <SOFTWARE_COMPONENT>, please renew or lose all packet data connectivity for your subscribers on <EXPIRY DATE>" every time anyone logs in to the management s/w when such a scenario becomes likely (e.g. for the last month). This should be a basic part of any s/w licensing feature.

      1. RegGuy1 Silver badge
        Facepalm

        why the fsck the s/w doesn't present a big flashing dialog

        Opps! Sorry that was me. I must have kicked the reminder machine that is under my desk, and I think I dislodged the network cable.

      2. Jellied Eel Silver badge

        Re: More detail

        More to the point is why the fsck the s/w doesn't present a big flashing dialog stating "Certificate about to expire for..

        .. failing to insert money. Back in the simpler days, you bought kit, it had software, you paid for a service contract that supported it, including access to software updates. Then along came software as a service, and new revenue streams. So instead of buying kit, you pay an annual rent or it can stop working.

        Which can (or should) factor into vendor selection given it can work out to a lot of money, especially if vendors want $$$ every time you add a device, or in some cases just add a new virtual circuit. Alcatel's NMS used to work on that model where you bought licence packs of points, and actions cost points. They weren't selected for a large network I worked on mainly for that reason. Nice kit, lousy business model.

        I'm a bit suprised that this happened. An expired cert should have been flagged as a critical risk, if that resulted in a network shutdown. Plus given $$$ for new licences, a sales bod should have been chasing for renewals.

        1. S4qFBxkFFg

          Re: More detail

          "I'm a bit suprised that this happened. An expired cert should have been flagged as a critical risk, if that resulted in a network shutdown."

          It should be considered almost as important as filing the annual accounts - what do the tax authorities respond with if someone forgets that?

      3. roblightbody

        Re: More detail

        As long as someone actually logs into the management console... or is looking at the notitication alert emails that its sending out.... thats if someone has actually configured it to send out emails....

        1. Doctor Syntax Silver badge

          Re: More detail

          "thats if someone has actually configured it to send out emails...."

          And if the recipient of the emails is still there.

          It's easy enough to set up a warning system. Protecting that warning system against the ravages of management changes is a different matter and almost certainly outside the powers of whoever set it up. If you were the one who was the designated recipient of the email and you've just been booted out of the job are you going to be in a mood to warn whoever did the booting that that particular mail box needs to be monitored? Is the booter even going to listen if you did? And will the booter get booted out in the next bout of changes?

          There needs to be personal responsibility on those making such changes to ensure that everything like this gets covered under the new organisation. HMG has woken up to the fact that national infrastructure needs to be protected even when it's in private hands. Maybe that protection should extend to personal sanctions on those involved, even up to CEOs and board members. Make them sweat a little. After a few big personal fines or gaol sentences businesses would become a little less cavalier about reorganisations and outsourcing.

          1. Anonymous Coward
            Anonymous Coward

            "And if the recipient of the emails is still there."

            I hope nobody really uses emails for that anymore - but for small networks. What are SNMP and all those expensive network monitoring systems for?? Big red lights should appear besides any device which have certificates about to expire. It's akin having and hardware components about to fail. You get proactive SMART alerts, but nothing about certificates...

      4. JohnG

        Re: More detail

        "More to the point is why the fsck the s/w doesn't present a big flashing dialog stating "Certificate about to expire for ...."

        Perhaps this was the responsibility of people amongst the 18,000 laid off by Ericsson in the last year.

    2. PerlyKing
      Joke

      Re: One of them needs a huge slap

      Even better, a "limited number" of slaps. Or maybe that's just the PR department ;-)

    3. Anonymous Coward
      Anonymous Coward

      Re: More detail

      was thinking the same, is it Ericsson who install and manage it or O2, you'd kind of think O2 would have noticed via SCCM or some other monitoring tool that it is going to expire

  5. Anonymous Coward
    Anonymous Coward

    Hey ...

    My old-skool Nokia 3310 was working just fine.

    O2? Nah, not me! lolol

    1. MrMerrymaker

      Re: Hey ...

      Who cares then? You're not relevant!

    2. JimboSmith Silver badge

      Re: Hey ...

      Couple on the table next to me at lunch didn't have any service on their phones. Not to be outdone by Sky Mobile not working they switched to their O2 backup sims. Sadly that wasn't working for them either and therefore "Every network must be down at the moment not just Sky!" They were most amazed when I received a call.........

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like