So why was there no redundancy in the internet link?
London's Gatwick Airport flies back to the future as screens fail
London Gatwick Airport’s shiny new cloud-based flight information display system had a hard landing this morning as its vision of the future was brought down to earth with a bump. While collecting the Cloud Project of the Year award at the Real IT awards in May, the airport proclaimed its new screens were "an innovative, cost …
-
-
-
Tuesday 21st August 2018 05:11 GMT Mike 'H'
But...VODAFONE...
Where the bloody hell was the LTE backup - 3 megabits on their Network must be child's play.
Point a directional YAGI or dish antenna towards an off-site tower to bypass on-site DAS/macros, likely afflicted with the same fibre cut, and bobs yer uncle.
Or maybe the airport should spend a bit more for a competing LTE provider for 3-megabit backup services...
-
Tuesday 21st August 2018 09:12 GMT Lotaresco
"Because oddly enough it doubles the cabling costs and that wouldn't do."
It really doesn't double the cabling costs. Pulling a multi-pair cable is a sensible precaution and if it is combined with the appropriate type of switch failover to an alternative pair is seamless. The switch will even notify that a pair has failed so that action can be taken by the SOA. The only difference in price is the cost of cable + switches which is minimal because labour is the big spend.
This is, quite frankly, poor practice on Vodafone's part.
-
Tuesday 21st August 2018 10:12 GMT ButlerInstitute
That wouldn't really be redundancy.
"Pulling a multi-pair cable"
But then you've got two cores in a single cable, thus failing to provide any resilience when a digger goes through the cable. Ok for a fault with the cable/core itself maybe, but not for a physical break.
Your alternative core needs to come into the site via a different route, So wouldn't be cut by the same digger. See for example major BBC facilities where there are redundant power and signal cables coming in from opposite sides of the site.
-
Tuesday 21st August 2018 12:37 GMT qwertyuiop
Two failures?
"...It really doesn't double the cabling costs. Pulling a multi-pair cable is a sensible precaution..."
Ah, I see. We increase resilience by using a multi-pair cable but immediately reduce it by using... a multi-pair cable. When the digger goes through the cable it doesn't matter how many pairs are in it! (and yes, I know that in this case it wasn't a digger)
The only way to truly increase resilience is to have two cables which arrive by completely different routes - and that will increase the cost.
-
Tuesday 21st August 2018 14:17 GMT Anonymous Coward
Ummmm.. yeah, it does. Multi-pair cables don't help if the cable's cut (since all the pairs are cut too). If you want to be truly fault-tolerant, you need a second cable, geographically isolated, going preferable to a second ISP but in any case at least to a different access point for your ISP.
-
Wednesday 22nd August 2018 14:44 GMT Mark Dempster
>It really doesn't double the cabling costs. Pulling a multi-pair cable is a sensible precaution and if it is combined with the appropriate type of switch failover to an alternative pair is seamless. The switch will even notify that a pair has failed so that action can be taken by the SOA. The only difference in price is the cost of cable + switches which is minimal because labour is the big spend.<
That probably wouldn't have prevented the issue, though, as pulling 2 cables from the same location to the same destination means they're separated by millimeters at best - so both would have been cut through at the same time.
If you're going to this trouble (and for something so important why wouldn't you?) then you need 2 completely different runs of cable, preferably from 2 different providers, coming into the building at different locations.
-
-
-
-
Monday 20th August 2018 12:55 GMT yoganmahew
Re: "no redundancy in the internet link"
Outages happen. The question for me is why there was no local cache? It would have grown stale over time, but a well installed local cache with a GUI for updates could put everything into manual mode with zero impact.
Is this the Internet of Tripe future? One failing link and your IoT belt unbuckles and your trousers fall down exposing your single-point-of-failure-arse?
-
-
-
Monday 20th August 2018 14:27 GMT Flakk
Re: "no redundancy in the internet link"
This tiny whiteboard is the only departures information in Gatwick Airport right now; every screen is offline. Utter chaos. This is a signature flourish at the end of a short trip that’s been full of reminders of how badly the UK’s infrastructure is crumbling. pic.twitter.com/6r7CDVheLf
— Rob Fahey (@robfahey) August 20, 2018
To be sure, that the info boards went down due to a network outage is pretty bad. But utter chaos? Really? It seems to me that any traveler who was inconvenienced by this outage for more than a minute has failed to understand the potential of the smartphone they are holding. You can absolutely use it to whine and wail on Twitter, or... I don't know... maybe look up the gate information on Gatwick's website? Failing that, maybe the airline's website?
-
Monday 20th August 2018 19:44 GMT JohnG
Re: "no redundancy in the internet link"
"...the smartphone they are holding. You can absolutely use it to whine and wail on Twitter, or... I don't know... maybe look up the gate information on Gatwick's website?"
Reports elsewhere indicated that updates to Gatwick's mobile app were affected by the same fibre outage. It seems that the design was engineered to fail comprehensively.
-
Monday 20th August 2018 20:09 GMT David 45
Re: "no redundancy in the internet link"
Potential of a smart phone? No earthly good if the information is not there to start with. I was there today, dropping off passengers, then attempting to pick up others. Website arrivals flight information was sketchy, with some flights missing or no information against the flight numbers (so no use for one's shiny smart phone there!). Gatwick's auto. phone information system just went dead the second I entered the flight number (so no use for one's shiny smart phone there!). I had no idea of the status of my incoming passengers' flight and felt that Gatwick's main number would probably be inundated, so I didn't bother trying that (so no use for one's shiny smart phone there!). Arrivals concourse info. screens were also all over the place, with my flight number also not showing on there at all. My passengers also said that the baggage reclaim section was also not working. The main complaint I have is that there was no information whatsoever in arrivals that there was a problem - not even any public address announcements and certainly no whiteboards. There were lots of baffled-looking folk with furrowed brows looking at useless screens, obviously wondering what the devil was going on. In fact, as far as I could see, the only information that something was amiss was a rider on the flight information website saying that information screens were not working properly. A bit more communication (in more ways than one!) wouldn't have gone far amiss.
-
Monday 20th August 2018 21:48 GMT Anonymous Coward
Re: "no redundancy in the internet link"
use a proper system like flightradar24 for any inbound flight they have real time up until the plane gets to the gate. For outbound the schedule will show that minus any delays.
There ar millions of ways to fins out whats happening without the airport on screen system if you travel enough you will soon get the hang of it
-
-
Monday 20th August 2018 22:56 GMT tin 2
Re: "no redundancy in the internet link"
"It seems to me that any traveler who was inconvenienced by this outage for more than a minute has failed to understand the potential of the smartphone they are holding."
Seems that information was not making it out of the airport to whatever cloudy goodness was updating the website and app either.
-
Tuesday 21st August 2018 09:17 GMT Lotaresco
Re: "no redundancy in the internet link"
"maybe look up the gate information on Gatwick's website? "
Have you tried to do that? Good luck trying it. If you're lucky you'll get departure gate information in time to watch your flight depart. If it's working as usual you will get the information 24 hours later.
I have the Gatwick app. It's never told me a gate number before the flight has departed.
-
Tuesday 21st August 2018 08:00 GMT Hans 1
Re: "no redundancy in the internet link"
It is pure and utter design failure.
Why have 100's of systems download flight information from the cloud ? <---- that is already brain-dead, somebody in the design team has never heard of multiplications!
No resilient internet connection ? <---- that is brain-dead
Caching is not the answer to a fundamental failure of resilient network design.
It reduces internet traffic significantly and buys you time for the connection fail-over..
-
Tuesday 21st August 2018 08:30 GMT Anonymous Coward
Re: "no redundancy in the internet link"
Welcome to the future, at least the future if the current "Cloud" fad continues.
Single points of failures all round.
Someone took a digger to your fibre? Bad luck. Maybe your backup link will take up the slack, but as these are usually lesser specification it will soon grind to a halt.
Someone took the cloud server down? No local caching or redundancy? Bad luck. Maybe someone will turn it off and on again soon. Maybe the script kiddies will get bored and stop DDOSing it.
Ironically the internet was developed to survive a nuclear war.
-
Tuesday 21st August 2018 13:17 GMT jmch
Re: "no redundancy in the internet link"
"Welcome to the future, at least the future if the current "Cloud" fad continues."
the thing is, a properly implemented cloud solution gives many benefits one of which is redundancy. Except management only got the memo where cloud = cost savings and don't realise that you still need to spend money on redundant network links. Every piece of the puzzle has to have redundancy, not only the cloudy bit
-
Tuesday 21st August 2018 18:58 GMT Martin Gregorie
Re: "no redundancy in the internet link"
One thing nobody seems to have forgotten - BT and other wonderful network providers currently operating in the UK have been known to engineer their own single point of failure. It happens this way:
- The system design team specifies a disaster recovery site and a high speed connection to it
- Their network design requires separate dual redundant links from the operations centre (LGW in this case) to the main ops site and to the disaster recovery site via at least two paths which are required to leave the building via separate ducts and then follow different routes.
- These specs get handed to the network provider, whose contractors promptly ignore all the fancy separate routing details and put all the cables through a single duct so they can trouser all the money they saved by skipping all that costly separate routing nonsense.
- The local council puts a digger through the cable duct....
-
-
-
-
Tuesday 21st August 2018 09:15 GMT Lotaresco
Re: "no redundancy in the internet link"
"The question for me is why there was no local cache? It would have grown stale over time"
It's an Arrivals and Departures system. The data grows stale in no more than a couple of minutes. A local cache doesn't really help. What is needed is resilient comms and that is standard provision for systems like this. There should be no SPOFs in a real time system.
-
Wednesday 22nd August 2018 07:59 GMT yoganmahew
Re: "no redundancy in the internet link"
@Lotaresco
"It's an Arrivals and Departures system. The data grows stale in no more than a couple of minutes. A local cache doesn't really help. "
Not really, the scheduled departure and arrival times are well known days in advance. The gates are usually well known, but at lest could be manually updated (so at least people stand a chance of finding their gate). The amount of data that tranmits by FLIFO for FIDS updates is vanishingly small, being essentially designed in the 1960s. Never mind LTE, you could run it on dial-up...
-
-
-
-
Tuesday 21st August 2018 09:54 GMT bpfh
Re: "no redundancy in the internet link"
So Gatwick generates the raw departures/arrivals data, posts that data to somebody else’s computer, then the screens pull that back to Gatwick. And it only uses 3 megabits...
So 2 questions:
0. Did they really need Somebody Else’s Computer to do this (especially if the computers that run this are still there, but now offsite - out of sight out of mind?)
1. 3 megabits... cut fibre... nobody thought of setting their mobile phone into WiFi hotspot mode - or dare I say it, backup 3/4G router ? Or is mobile internet that crap there?
-
Wednesday 22nd August 2018 08:03 GMT yoganmahew
Re: "no redundancy in the internet link"
@bpfh
"DId they reall need someone else's computer"
No, but, generally you stick the flight information data on a server somewhere so you can access it from multiple sources. Of course, if you build that as PUBSUB and there was a local server serving coax to the airport... (why coax? Coax can be fixed by anyone with pliers and a piece of tape... Bring back coax!).
It's a bit weird, that they are having problems still suggests it's not just a fibre cut?
-
Wednesday 22nd August 2018 09:23 GMT defiler
Re: "no redundancy in the internet link"
I'm just going to lob this on the end here, since everyone's screaming about having a second cable (expense of installation, potential proximity to first), and others are screaming about having a cellular backup (potential proximity to cable because Vodafone).
Nobody's saying satellite link. If all you're looking for is 3Mb/sec, it's not expensive. Oh noes - 200ms latency? Who cares? If the power stays up, the data stays up. And if the power drops, the airport is closed anyway.
-
-
-
Monday 20th August 2018 14:35 GMT Anonymous Coward
Situation Normal: Outsourced Data
When LGW was run and managed by BAA, BAA IT provided the information screens in co-operation with CAA/NATS in the tower.
Now, with GIP owning the Airport, VodaFone providing the data and the tower outsourced to a German company with an unfinished website - hardly surprising really....
Too many layers of outsourcing....too cheaply....
-
-
Monday 20th August 2018 11:53 GMT Anonymous Coward
According to http://www.mediacentre.gatwickairport.com/press-releases/2018/18_05_18_it_awards.aspx
"We have also just future proofed our entire IT network for the next decade so that we can take advantage of the latest technologies, while also making the network more resilient and tolerant to disruption."
Bullsh*t...
Biting the hand that feeds IT
