back to article Ah, British summer. The sun is shining, the birds are singing, the internet is on the fritz

As the big red ball in the sky continued to shine on the UK, internet providers decided to have a bit of a lie-down, with Sky, TalkTalk and leased line specialist M24Seven all taking a turn on the sun-lounger. Sky and TalkTalk's problems began in the early hours of 17 July as night-owls in the Portsmouth and Southampton area …

  1. Bavaria Blu

    Big red ball?

    From where I'm sitting its a white ball, possible yellow and never red. Are you thinking of Japan? The sun is red there, as you can see from the flag.

    1. John G Imrie

      Re: Big red ball?

      If you are looking at it through the smoke around Manchester it might look red

      1. Aqua Marina

        Re: Big red ball?

        I'm actually surprised that the 2 week long fire on Winter Hill didn't knock out any communications. Here's a pic showing how close the fire got to the transmission tower goo.gl/ajskPr

        For those not from Manchester, an area of about 4sq miles of parched moorland was burning for a couple of weeks until it rained last week resulting in the hill disappearing in a cloud of smokey mist. I drove up as close as the road blocks when it started raining, it smelt like a damp barbeque!

        1. BongoJoe

          Re: Big red ball?

          I was watching that fire not too long ago. We had heaved to in our motorhome to a farm site for a few days and we were watching the fire approach those towers with great interest. And the key in the ignition...

          1. Orv Silver badge

            Re: Big red ball?

            Last year the Thomas Fire in SoCal played havoc with the power to the Santa Barbara area, as there's only one transmission line and the fire was burning right along its path. The problem wasn't so much direct damage to the wires, it was that the smoke particles and heat reduced the ionization threshold of the air and caused the line to expand and sag, forcing them to cut the power to avoid catastrophic arcs.

      2. Anonymous Coward
        Anonymous Coward

        Re: Big red ball? (Smoke around Manchester)

        ooops, I thought that the red sun seen through “smoke around Manchester” comment was a Lowry reference at first(!).

        I sincerely hope that Manchester (and people I it) has not been too frazzled by the wildfires.

    2. Anonymous Coward
      Anonymous Coward

      Re: Big red ball?

      Never red? Nope, the description will eventually be correct, just wait a little while.

      OK, you'll have to wait quite a long* time, but it may just possibly be long enough for certain providers to understand what Customer Service actually means.

      *: 5 or 6 billion years

      1. Alan Brown Silver badge

        Re: Big red ball?

        "long enough for certain providers to understand what Customer Service actually means."

        They already know: https://www.youtube.com/watch?v=dytZY74bOM4

      2. John Brown (no body) Silver badge

        Re: Big red ball?

        "Never red? Nope, the description will eventually be correct, just wait a little while.

        OK, you'll have to wait quite a long* time,"

        Nah, you only have to wait a few hours for almost sunset then goes a lovely red if the weather is right.

    3. defiler

      Re: Big red ball?

      It was magnificently red when it set last night. Admittedly it was off the coast of Spain, heading North at >30000ft. However, it was the nicest past of a Ryanair flight.

    4. Captain Hogwash

      Re: Japan?

      Or Bangladesh?

    5. hplasm
      Alien

      Re: Big red ball?

      Brexit means Brexit - to Betelguese V...

      - Teresa May Beeblebrox

  2. b0llchit Silver badge
    Facepalm

    Pick two from the trinity

    Good, Cheap, Fast, pick two, you can't have all three.

    People should not complain about a service going down. They pay the "right" amount for their service and always expect a trinity when they just pay for the cheap fast one. So, it ain't no good.

    1. ArrZarr Silver badge
      Happy

      Re: Pick two from the trinity

      When you pay for Good and Fast, however, this is a bit annoying ;)

    2. Anonymous Coward
      Anonymous Coward

      Re: Pick two from the trinity

      You can bet sky will claim compensation from whomever broke the fibre and a tax loss, some of that should be returned to the customer given that sky are not providing the service they are getting paid twice for everytime it is someone elses fault.

      1. Chronos
        Flame

        Re: Pick two from the trinity

        You can bet sky will claim compensation

        As they damned well should. Anyone, anyone who cuts a cable by being a blithering, incompetent idiot should be sued to oblivion - as should these utter arseholes who wait until a road has been resurfaced and then decide they need to stick in a pipe¹ or a cable, whereupon they replace it in exactly the same manner that necessitated the resurfacing work in the first place.

        Bastards. Would it be so difficult to have a gazette notice akin to "We're resurfacing Shit Alley on the fifteenth of September, anyone wishing to lay pipes and cables needs to get in now because we're entering a two-year sod off we've just fixed it period."

        ¹ British sodding Gas, I'm looking at you. Our road has been resurfaced once in the 20 years I've lived here and fuck me sideways with a ten foot dildo if, within a fortnight of it finally being done, you twats weren't here making it lumpy again.

        1. Alan Brown Silver badge

          Re: Pick two from the trinity

          > Bastards. Would it be so difficult to have a gazette notice akin to "We're resurfacing Shit Alley on the fifteenth of September, anyone wishing to lay pipes and cables needs to get in now because we're entering a two-year sod off we've just fixed it period."

          My local council did that for a major road about 30 years ago, relaying and rebedding a 5 mile stretch of road and footpath. They interacted with _ALL_ the utilities to ensure that noone would dig up anything for at least a decade afterwards.

          2 weeks after the job was completed the telco ripped up the newly laid footpaths and killed all the newly planted trees along the entire length of the road to install ducting - apparently because someone in head office couldn't be arsed to read the documentation for 10 years and then misread the end date as the start date. They got served with a very large bill (all the utilities had signed off on an undertaking not to dig) and a court order to make it as good as it was before they started.

        2. MonkeyCee

          Re: Pick two from the trinity

          "Bastards. Would it be so difficult to have a gazette notice "

          My Dad was working on a project for pretty much exactly this, nicknamed "hole in the road" program. The idea was that utilities could see where other people's pipes are buried, and so either pick a least destructive path, a careful excavation where the other pipes are or even co-ordinated digging up stuff.

          No idea what came of it, but it turned out that the gas, water and some of the lines companies had only a very vague idea where certain pipes went.

          1. Anonymous Coward
            Anonymous Coward

            Re: Pick two from the trinity

            No idea what came of it, but it turned out that the gas, water and some of the lines companies had only a very vague idea where certain pipes went.

            This remains the case (I used to work for this sort of utility). When you ask "Do you know EXACTLY where your pipes are?" you will certainly get a "YES!, YESSIR! YES!" answer, accompanied by some extensive horseshit about a fantasmagorically accurate GIS. If you persist in your questioning enough, or merely dig the sodding road up, you'll find out that the GIS is accurate to a level that might be termed "reasonable guesswork".

            1. Orv Silver badge

              Re: Pick two from the trinity

              Hence why the end stage of pre-digging prep tends to involve sending some guys 'round with metal detectors and spray paint.

              Projects in the US to nail this data down more comprehensively stalled after 9/11 due to security concerns; it seems that having one central, accessible database with the locations of all the vital infrastructure was maybe not the best idea. One poor bastard had his PhD dissertation spiked by the NSA for that reason -- he'd mapped out all the Internet backbone cables. Left him in a bit of a bind, since you can't be awarded a PhD without publishing your work.

            2. Chronos

              Re: Pick two from the trinity

              If you persist in your questioning enough, or merely dig the sodding road up, you'll find out that the GIS is accurate to a level that might be termed "reasonable guesswork".

              i.e. "turn left at the Red Lion, Shit Alley is the second on the right, assuming you're in the right $VILLAGE of which there are three in relatively close proximity with the same name (Wales). Look for the shittiest excuse for a road surface in human history, there's a good chance it's ours."

              One wonders if the old Ford Cortina would come in handy here. As many know, if the rear spring bushes were over a fortnight old, the 'tina would follow every single trench dug within the past century, resurfaced or not, like it was on rails. Made your bottom go like a rabbit's nose but could be quite useful for forgetful utility companies.

              1. Anonymous Coward
                Anonymous Coward

                Re: Pick two from the trinity

                Made your bottom go like a rabbit's nose but could be quite useful for forgetful utility companies.

                Not really forgetful - when most of the pipes were laid there was neither need nor the technology to record with much accuracy. Even now, it is all very well mapping to high precision, but that's pointless when the "resolution" of digging through tarmac and sub-base is about +/- 4 inches at best.

                The fundamental problem is that infrastructure is expected to last, so it has invariably been shoved under roads for a range of good reasons when you plan to leave it undisturbed for a few decades or more, but at this point in history we're needing to replace or repair a lot of water pipes, gas pipes, sewers, electricity distribution cables and telecoms ducting, largely as a function of age, plus some relatively recent changes in infrastructure performance expectations, plus some really stupid ideas - in particular the EU decision to permit heavier lorry weights, which has caused massive cumulative damage in the UK due to the increased axle weights on all roads that were never designed or made up for those loads.

                1. Anonymous Coward
                  Anonymous Coward

                  Re: Pick two from the trinity

                  The only other thing I would add is that the pipes don't always remain exactly where they were buried.

                  Add a little water and traffic vibrations, combined with a mix of soil consistencies and pipes/conduit can wander off those wonderfully "accurate" GIS...

              2. CrazyOldCatMan Silver badge

                Re: Pick two from the trinity

                One wonders if the old Ford Cortina would come in handy here

                "Cure worse than disease" springs to mind. I had a Mk3 Cortina for a while in my yoof. I think it's probably the worst car I've ever driven - and I've driven my wife's Morris Minor..

                Unlike yours, mine had a suspension that vaguely approximated the stiffness of blancmange. Forget cornering at anything vaguely approaching speed since turning the wheel with any degree of rapidity left the bodywork wallowing around in a sort-of-average-direction.

                It's the only car that I've ever skidded unintentionally.

                1. Chronos
                  Mushroom

                  Re: Pick two from the trinity

                  Unlike yours, mine had a suspension that vaguely approximated the stiffness of blancmange.

                  On the Cortina I had in mind, that was quite an accurate description of the brake discs. It wasn't mine, I hasten to add. The one and only Ford I would ever consider owning is the long-gone Escort MkI Mexico.

                  Vauxhall took up the gage in the battle to become the manufacturer most likely to stop fitting steering wheels with the Vectra. So much understeer that you may as well give up cornering and become a Roman.

                  and I've driven my wife's Morris Minor..

                  Amazed it survived with that Lucas regulator. That's why most of them were positive earth, so the charged electrolyte that spewed out of the battery when the regulator welded itself shut was repelled, to some extent, and the cause of all the rot wasn't quite so obvious as the battery dropping to the floor through the bran flakes it had created where once was metal...

                  Can you imagine Lucas of old wiring up one of these posh milk floats stuffed with li-ion cells? Icon says it all.

        3. CrazyOldCatMan Silver badge

          Re: Pick two from the trinity

          Anyone, anyone who cuts a cable by being a blithering, incompetent idiot should be sued to oblivion

          To be fair to the digger driver - a lot of the conduits (especially out in the countryside) are not particularly well mapped. It's entirely possible that whatever GIS system they were using didn't have the conduit marked on it.

          Unlike the conduits that carry our fibre here at work - so there was no excuse for the building contractor nearby when they dropped a load of spoil onto the fibre conduit access hatch and crushed the cables. And then wouldn't move the spoil to enable BT to fix the fibre.

          So say that relations got a tad frosty is a bit of an understatement. Words like "lawyers" and "compensation" were uttered, even though the building contractor is also our landlord..

          1. Chronos

            Re: Pick two from the trinity

            To be fair to the digger driver - a lot of the conduits (especially out in the countryside) are not particularly well mapped. It's entirely possible that whatever GIS system they were using didn't have the conduit marked on it.

            Oh $DEITY no, I wasn't thinking of suing some poor bugger trying to do his or her job. I was thinking more of the people with extremely thin spectacles on three times their salary who hang around in offices with titles such as "planning officer" or "civil engineer" who can't be arsed to even look for potential cock-ups because that would eat into "looking sophisticated drinking civet-turd coffee" time, along with the companies that allow them to do so with impunity while Ms digger driver is getting verbal abuse from the general public simply because she lacked the information she needed to avoid that duct.

            In fact, Ms digger driver should get a bit of compo, too, for being left in the line of fire.

            1. Orv Silver badge

              Re: Pick two from the trinity

              I had a brief stint with a railroad signal department, and what really made them sweat bullets is directional boring. In theory it's a great time saver -- instead of digging up a road, just run a small tunneling device under it that you can steer. In reality, you're flying blind through dirt with very little idea of what you might hit on the way. I believe in the end they decided, over some objections by the union, to just contract out all their directional boring work so that someone else would bear the liability.

  3. Alan J. Wylie

    Fibre break(s) near Manchester, too

    Zen: Fibre Cable break impacting Bury Exchange

    Exa: At approximately 10:22am one of our core fibre links from London Telehouse to Manchester experienced total failure.

    Bury is north of central Manchester, so these may be separate incidents.

    1. hplasm
      Coat

      Re: Fibre break(s) near Manchester, too

      "Bury is north of central Manchester, so these may be separate incidents."

      Or a break in the middle...

      * Go north Fibre, then sharp right at Bury. No- not that sharp!! -*snap*. *

      *bugger.*

  4. mark l 2 Silver badge

    "as a small business, such an outage could leave them out of pocket to the tune of £5,000."

    I would suggest that if you were so reliant on internet that you could be loosing 5 grand for an outage, you should have some sort of disaster recovery plan for eventualities where the internet might get borked. It could be as simple as having an old pay as you go phone in a draw which can be used as a WiFi hotspot to be able to carry on operating until your main connection is restored.

    1. Anonymous Coward
      Anonymous Coward

      Depends, can you make hundreds of VoIP calls over PAYG mobile internet?

      1. Anonymous Coward
        Anonymous Coward

        Probably not, but if you lose 5k for an hour or so outage then you make sure you've got supplier redundancy and you've made sure that fibre routes don't pinch etc. it's not uncommon for a JCB to cut through fibres and it take hours to fix.

        1. Arthur the cat Silver badge

          if you lose 5k for an hour or so outage then you make sure you've got supplier redundancy

          Except their sales people usually don't know where the fibre runs, and then when the JCB hits you find at best both your suppliers had separate fibre, but in the same duct. There are only so many ducts in any area and they run in obvious directions. A wireless backup could help, but where does the backhaul run? In the same geographically obvious duct. Ultimately a satellite link avoids most JCB related problems, but good luck getting VoIP to work on that.

    2. Dr. G. Freeman

      I remember during the dark days of working the BT tech helldesk, getting "I'm losing millions of pounds here because you broke my internet" at least twice a shift, from launderettes, corner shops, and other such businesses.

  5. Jason Bloomberg Silver badge
    Mushroom

    Talking of bright things in the sky

    The Internet. Supposedly resilient enough to keep itself going in the event of nuclear war. In practice unable to keep granny on-line when confronted with the mighty power of the JCB.

    1. Alan Brown Silver badge

      Re: Talking of bright things in the sky

      "In practice unable to keep granny on-line when confronted with the mighty power of the JCB."

      In my telco days the (friendly?) rivalry between microwave and fibre techs was summarised by calling this "back-hoe fading".

      In later days it's proven to be a very useful way of getting spammy datacenters offline.

      1. MonkeyCee

        Re: Talking of bright things in the sky

        " rivalry between microwave and fibre techs"

        One site I managed had the main link by microwave, with a backup DSL line.

        During one summer (education, so hardly any users on site thankfully) the DSL line got cut about 8am, due to the tender caress of the back-hoe. Also took out the none VOIP phone lines.

        Then around morning tea the microwave link went down. After a short investigation by the provider (telescope from their office building) they diagnosed the cause as "some prick in a cherry picker".

        So there you go, the cherry picker is the backhoe for microwave :)

        1. Anonymous Coward
          Anonymous Coward

          Re: Talking of bright things in the sky

          So there you go, the cherry picker is the backhoe for microwave :)

          Looking on the bright side, the bloke in the cherry picker's bucket won't be having any further offspring. Presumably that lucky fellow won a Darwin Award whilst remaining alive?

          1. John Brown (no body) Silver badge

            Re: Talking of bright things in the sky

            "Looking on the bright side, the bloke in the cherry picker's bucket won't be having any further offspring. Presumably that lucky fellow won a Darwin Award whilst remaining alive?"

            If that were possible, there'd a be a line/cone of dead birds on the ground radiating out from the transmitters. Cherry picker bloke would have to be right in front of and close to the transmitter for an extended period to come to harm.

            1. Orv Silver badge

              Re: Talking of bright things in the sky

              Cherry picker bloke would have to be right in front of and close to the transmitter for an extended period to come to harm.

              A friend of mine used to work on military radar systems. Those really *do* have enough power to cook you in short order, and the wavelengths in use tend to resonate in some pretty important body structures. Eyeballs, for example. They were informed that if the transmitter was accidentally switched on, their first hint would be dry eyes due to their tears evaporating, and the recommended action was to immediately step off the tower. Whatever injuries they got from being caught by their safety harnesses would be far easier to repair than cooked eyeballs.

    2. Orv Silver badge

      Re: Talking of bright things in the sky

      To be fair, the military protects its links rather more carefully than your average Internet provider. I've heard tales of fiber encased in two-foot-thick concrete. For microwave links, the solution was to use some extremely robust antennas: http://long-lines.net/places-routes/MD01/020719B-11.jpg

  6. YourNameHere

    Dedicated lines

    If you buy a dedicated line, you get get a dedicated line. Singular. So if it gets cut, don's act surprised by the results.... :)

  7. Slx

    The internet's effectively a mesh, but a mesh is only a mesh if it has more than one fibre route.

    It looks like they had a single point of failure and basically what that amounts to is cheap infrastructure.

    1. Orv Silver badge

      Even if you pay for redundancy it's not unlikely that your two connections with both end up running along the same highway, power line, or railroad corridor. There's only so many places it's cost-effective to put cable.

      1. Anonymous Coward
        Anonymous Coward

        At least for my home setup, I have a RPi with a 3G phone on it acting as a failover gateway in the event my main router/line goes down I just stick some credit on the SIM and I'm back online. There really is no reason to complain for people using consumer lines for small business, there are plenty of cheap solutions available if you're willing to pay an IT guy to set them up.

        The above is just redundancy for a nerd, no business running here. I had a 30 day Internet outage a while back and figured I might as well keep the setup, it uses failover DHCP and advertises both gateways with priority metrics so clients auto-failover once packets start dropping from the primary gateway.

        Granted, while in failover mode, the 3G data price is ridiculous, but since most outages are less than a few hours, really it works out at a £10 top-up per outage. Total setup cost: ~£50.

        1. Orv Silver badge

          That's clever, and will fix local outages fine, but are you sure your 3G provider doesn't use the same fiber for backhaul that your ISP does? I've lived in places where a fiber cut took down not only Internet for a college campus, but phone service for the entire town, before.

          In that case the mesh concept did work for campus -- sort of. They had another link to a satellite campus, which in turn happened to have another backbone link. Problem was this resulting in the entire traffic load of a T3 trying to go down a single skinny ATM line, so in practice it was almost totally unusable. Made for some interesting traceroutes, though.

          1. Anonymous Coward
            Anonymous Coward

            I remember way back when ATT had a fiber ring cut. IT took out 911/internet and version cell phone towers. Turns out ATT was one of two intent back bone in that area . Good luck when a major fiber node in a rural area gets cut.

      2. Pen-y-gors

        @Orv

        The whole theory of an InterNET is that it's a network. No node should be connected by a single link. So an area like 'Portsmouth and Southampton' should have a fibre link running northwards, but also another running east towards Brighton and thence to London and probably one running west to connect up to Bristol or Swindon and then back up the M4.

        1. CrazyOldCatMan Silver badge

          one running west to connect up to Bristol or Swindon

          Um.. Swindon isn't very much west of Portsmouth..

          1. M Mouse

            "Swindon isn't very much west of Portsmouth"

            Hove is around 50 miles east of Portsmouth.

            Swindon is over 50 miles west of Southampton.

            What actual locations were you thinking about?

            It seemed quite reasonable to me to indicate such places as "near but not too near" as separate nodes for routing traffic another way if some disruption takes place. More awkward locations would be those such as Penzance where a single connection is most likely.

      3. Anonymous Coward
        Anonymous Coward

        Just as bad is when you get a a leased line from ATT and sprint only to find out later the Sprint is running over ATT for the last mile .

  8. Teiwaz

    Talk Talk went down?

    So the guy on the exercise bike that powers the system fell off then?

    1. Slx

      Re: Talk Talk went down?

      At least it's environmentally friendly - the whole network can be run for several hours on a sandwich.

  9. Frederic Bloggs

    Guerrilla Marketing?

    More like (angry) Gorilla Marketing

  10. Steve Jackson

    You do the math

    Come on, M22.75Seven unless you have invented the 25 hour day.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like