BT blames 'faulty router' for mega outage. Did they try turning it off and on again? BT has blamed a faulty router for knocking its network offline yesterday, leaving hundreds of thousands of customers without the internet. The telecoms giant apologised for the failure, which began at around 2pm yesterday afternoon. Customers across the country were unable to get online, with reports of the outage affecting …
"I've no idea how good or bad their HomeHubs are"
They're s**t.
I had the HH3 (or 4) when I moved to them for regular ADSL, which I replaced with a NetGear somethignorother fairly quickly when I realised how flaky the BT kit was. Then, having lost all patience with the HH5 I got from them for the Infinity service, I've just replaced that with a NetGear Nighthawk.
Little wrong with HH5 for what it is, router "random" reboots are caused by BT sending instructions over the WAN instructing it to do so
Individual consumers simply get fobbed off as dumb or replaced router and no way of getting behind the issue to any technical team..
BT HH seems to = outage-as-a-service despite the routed itself being adequate.
@ Anthony Hegedus
That's funny.
I have a collection of BT-supplied routers going back about a dozen years to when I switched to wireless. All of them are still in perfect working order and the only lengthy hiccup I have experienced in that time (2 days out) was tracked down to a problem at the local exchange. The comment sounds like those that I see about Windows 10 - I hate BT because they are BT.
"I hate BT because they are BT."
Most people hate BT because of what those people have learned from experience with BT.
"I have a collection of BT-supplied routers "
These broadband routers may have been BT-supplied, but prior to the days of HomeHubs and so on they almost certainly weren't BT-designed.
So thank BT for choosing a decent vendor's routers back then.
Nowadays, with HomeHub... let's not go there, OK.
"So thank BT for choosing a decent vendor's routers back then"
total bollox
Before the HomeHub 1, BT used a mix of routers and ADSL modems, mainly from Alcatel-Lucent (or whatever they were called that week) There was the green frog modem, replaced by the purple slug, which both had the same guts as the Alcatel 105
Total crap.
The frog was crap, I'll give you that, but what else was there in its day?
The best SoHo ADSL modem I've ever had was the BT-badged Voyager 2100 and its close relative the V2110. Good at doing what it needs to do, decent user interface, readily usable user-friendly diagnostics and reasonably detailed troubleshooting info for when something breaks, *and* manageable by SNMP, including the DSL Line Statistics MIB. Hadn't seen anything close to it since the D-Link DSL604+. The Voyager 2100 was a GPL-based router (though BT forgot the implications of that detail initially) closely related to another vendor's product, probably a 3Com in a different box, though I forget.
I think the Voyager routers were a french Thomson-branded design, which was an earlier iteration of Alcatel. That unit has had so many names due to the various takeovers and selloffs from Alstom/Alsthom/Alsace-Thomson-Heuston/Roneo/Alcatel/Lucent.............
as regards ADSL modems - far superior to the frog and slug was the Fujitsu FX-310, a wonderful little device which had the benefit of when plugged into a phoneline it rebooted stale ADSL sessions automatically. I used to carry one just to clear bad sessions. Not something you hear of nowadays
"I think the Voyager routers were a french Thomson-branded design, which was an earlier iteration of Alcatel."
Some of them might well have been. Back in the day, I was far more familar than a user should be with the V21x0 and its internals, and it wasn't French.
"I used to carry [Fujitsu] just to clear bad sessions. Not something you hear of nowadays"
That brings back memories. Not necessarily good ones either. Thank you (?).
One faulty router brings the network down? Seems like a poorly designed and/or configured and/or monitored system to allow that to happen and to affect disparate parts of the UK for so long. However one faulty GPS satellite seems to have stopped DAB radio from being accessible to parts of the UK for several days, so maybe it's possible.
One faulty router brings the network down?
If it starts announcing gibberish instead of what it is supposed to announce as routing updates - why not.
There is bugger all protection at the routing protocol level for most internal gateway protocols. OSPF has none by design (no, do not talk to me about the admin weight hack in Cisco IOS the person who invented that should be shot), ISIS is not much better, iBGP is usually unfiltered as well and let's not even talk about various protos that used to be popular with with BT CTO like PBB and PBB-TE.
The solution in this case is to have good "view"/"analysis" of the routing protocol state and KILL the router from the wall switch right away to localize the failure. I am not going to comment on BT and either one of these.
> If it starts announcing gibberish instead of what it is supposed to announce as routing updates - why not.
Thank you for being the sensible one here. I was expecting the standard "BT are shit" comments. I've had bad experiences in former houses but the last two I've lived in Cheshire have had no problems for 6+ years. I'll give BT credit where it's due.
Would you not expect a company of BTs size to have multiple CCIE types on their books, with an incredibly high spec network that is well designed to cope with the network traffic of the UK? Unfortunately all it takes is bit rot somewhere down the line, and you're sending out spurious data. The above poster is spot on.
"Would you not expect a company of BTs size to have multiple CCIE types on their books, with an incredibly high spec network that is well designed to cope with the network traffic of the UK?"
No I wouldn't, based on the last few years, though obviously it might be a laudable goal. Nor would anyone else who has followed various BT network shenanigans e.g. those involving congestion in BTwholesale's backbone network, congestion which was repeatedly denied in public by the then "BT Chief Network Architect" around Feb 2014, and various similar foul ups. His name is public domain but out of misguided courtesy I won't post it here.
http://www.revk.uk/2014/02/stats-arent-facts.html
This post has been deleted by its author
"Backup router? but that would double the cost surely, and how often would we need it, really?"
14 or so days out of every 28, because you deliberately fail it over every few days, just to test the failover procedures still work as intended.
Though as has been noted elsewhere, there can be a world of difference between total failure of a network component, and same component going insane in a way which simple automated procedures don't quickly detect. Even so...
No need for it, Its Most likely Cisco kit and the Chinese will have used the backdoor to enter it and fix it for them by rolloing back to the version where they get copies of all the data going through.
You would expect that the NSA would have informed them they were going to do some software mods though......
I belive BT are using Huawei for the 21CN kit. And they are not exactly the sort of router you'd have at home.
Something like this I would guess....but those with more knowledge of BT backbone may know better.
http://e.huawei.com/uk/products/enterprise-networking/routers/ne/ne5000e
Dear BT... if you have single points of failure that can bring down hundreds of thousands of your customers then you need to fire the useless gimps who designed it, and employ someone who actually understands the concepts of resilience, high availability and possibly even DR.
Understandable that you had your support call centres flooded with calls but over the space of an hour I either got cut off immediately or an engaged tone. On the handful of occasions I did get in a queue you disconnected me to silence after 3m 30s. You are BT ffs! If you can't handle call volume correctly there's no hope for anyone.
Eventually you had a recorded message. Of course, by then world+dog had already worked out there was a major outage.
Single router ffs.
As I said elsewhere, myself and most people will accept the odd problem and this is only the second in 3 years. I am pissed off though that it was nigh on impossible to get information from the horses mouth. Bad, BT, bad.
"...
You need to read up on Spanning Tree..."
No I don't. I have networking specialists to do that side of it for me. Failing that there's always someone on el reg happy to prove they know more than someone else and to do it in the most condescending manner possible. You know, like you just did.
Delegation. Didn't learn that on any course, ITIL or otherwise.
ITIL...ITSM... all nicely wrapped up in arrogance... I smell a "consultant"
Oooh I know. .. I'll see yours and raise you a TOGAF!
People so needlessly aggressive as you are being and tryIng to prove endlessly they know more than random strangers online tend to be hiding behind the buzzwords and bravado.
Ladies and gentlemen we seem to have found Dick from the internet (see various Dilbert sketches). That or... do you by any chance work for Capita???? I suspect so given your complete lack of a sense of irony or humour.
Spanning tree is for switching (layer 2). Routers are for routing (layer 3) and don't use spanning tree in a network core since all their connected networks are usually point to point (no loops). They do use it at a layer 2 edge though yes, but a problem there would only affect that layer 2 domain.
... just tell them we had a router failure. It took us a while to try a new filter but that didn't fix it so we had to unplug all the phones and disconnect the extension cables and plug the router directly in to the wall port, you know, so that the only equipment on the line was the router itself, but don't tell them that while we were keeping everyone looking at the right hand, they didn't see the left hand plug in the network cable that it accidentally unplugged at the exchange.
And if they do find out, just blame an unknown third party because we've been forced to open up our premises to the competition.
@msknight and ummm ... did they try:
"Remember to reinstall Windows as we can't help you diagnose why your internet isn't working until you've done that. My call plan says you have to. You might be using your computer for things we're not aware of, so you have to reinstall it. Really you do."
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
