back to article Telecity fails with car park net rescue plan. In fact, things got worse. Again

Internet hub Telecity has failed to overcome the recent power fault responsible for frying computer hardware and denting UK internet traffic. The data centre and colocation centre provider's plan was to employ a static transfer switch to transfer load to a mobile DR system installed especially in the Sovereign House car park, …

  1. Red Bren

    They should be applauded

    A comprehensive disaster recovery exercise that includes an actual disaster...

    1. Warm Braw

      Re: They should be applauded

      Maybe they're just taking "lights out" to its logical conclusion...

    2. Anonymous Coward
      Anonymous Coward

      Re: They should be applauded

      Who needs ISIS to bring down the internet when we can do it ourselves...

    3. Greg J Preece

      Re: They should be applauded

      Or for managing to fry servers through a UPS. Impressive.

  2. Anonymous Coward
    Anonymous Coward

    Parse failure

    "engineers took the power out of phase producing a massive power spike that took out the B system UPS – a diesel-rotation uninterruptible power supply."

    I'm having trouble making sense of this sentence.

    Suggestions?

    1. Chris Miller

      Re: Parse failure

      When you're connecting or switching HV systems it's very important that their phases match, otherwise you get an instant transition from top to bottom of the waveform (a 'spike') that can wreck any sensitive electronics that isn't protected. And the correct phrase is Diesel Rotary UPS (DRUPS).

      1. PJF

        Re: Parse failure

        ... And the correct phrase is Diesel Rotary UPS (DRUPS).

        I don't know much about DRUPS, but, do they have "Synchronizing" lights/meters. I know larger commercial/military generators that can be paralleled, do have synco lights to help the engineer get them into phase before throwing the big switch.

        1. Dan Wilkie

          Re: Parse failure

          Usually most of the modern large ones have an automated system that would do it, you normally only use the manual synchometer (or whatever the hell you can call it) as a final manual resort. Though I've seen it set up on at least one class of ship where the manual system takes it's 5v from one of the automated systems, so if that fails you can't join the genny manually either...

          It's quite impressive if you bork it completely though - on my work experience they showed me a video of one on a warship that had been hooked up 180 deg out of phase and it ripped the generator of it's mountings...

          1. Anonymous Coward
            Anonymous Coward

            Re: syncometer

            Thanks for clarification.

            New BBC/OpenUniversity series "Power to the People" did an hour or so on Ferrrybridge power station last week, a former 2GW station soon to be 0GW but at the time of recording it had two units at ~400MW each. A brief segment covered bringing a 400MW unit back online using a neat looking 1960s sync-matching device. It was around the same age as most of the other visible control equipment.

            http://www.bbc.co.uk/iplayer/episode/b06q4z6j/power-to-the-people-1-keeping-the-lights-on

            and/or

            https://www.youtube.com/watch?v=JSoa6XKgJXo

            1. Anonymous Coward
              Anonymous Coward

              Re: syncometer

              Yes - the whole control room looked like it had once been the set for the TARDIS secondary control room in a Patrick Troughton episode of Dr. Who.

              The programme itself was a bit disappointing: at one point the talking head opened some cabinet doors pointed at some flashing lights and said "Look, flashing lights." And then closed the cabinet. No explanation whatsoever.

              Likewise the 'syncometer' - no explanation of how it works.

              Note to Beeb programme commissioners: dumbing down a programme won't get you more viewers if it's billed as being about a power station since only the 'geeks' will turn on anyway. All you have done is annoy those that did watch and weren't expecting a children's programme. And it's BBC4 FFS!

              1. Anonymous Coward
                Anonymous Coward

                Re: syncometer (Power to the People)

                " it's BBC4 FFS!"

                BBC4 and Open University co production, no less.

                I didn't get what I might have hoped for, but what I got was a lot better than I had feared, and thus was actually a pleasant surprise. Especially as a 400MW failure at Ferrybridge had been blamed elsehwere for the previous week's chaos on the grid (whereas the lack of 5GW+ of wind wasn't mentioned). The programme made it quite clear that the station was being run down, in particular that maintenance was being run down. It was not stated explicitly but it is obvious that an increase in unplanned outages should not be a surprise and it should not be a surprise if they last longer than would previously have been expected.

                With the limited insight provided, it looked very much as though the station was still largely manually operated, maybe hadn't even moved into the era of PLCs, never mind the era of SCADA and beyond.

                Hack that. you horrible hackers. Oh hang on, no need, it's closing next year anyway, like many UK coal fired power stations were even before last week's announcement, because the ongoing lack of investment (particularly in flue gas desulphurisation) means it doesn't meet the decade and a half old anti-pollution regulations.

                But that's another story for another article.

              2. Anonymous Coward
                Anonymous Coward

                Re: syncometer -Likewise the 'syncometer' - no explanation of how it works.

                The simplest "syncometer" is a bulb connected between the same phases of the two generators to be synchronised. When the phases are in opposition, the bulb is very bright (twice normal pk/pk volts). When they are in phase the voltage at both ends of the bulb is the same and it is out. However, the bulb might blow...

                The simplest syncometer, therefore, is just a full wave rectifier replacing the bulb and connected to a voltmeter.

                If the generators are 3 phase only one meter is needed because the relation between the phases is geometrical - if one is right all three must be right provided both generators are turning in the same direction.

                1. Will 30

                  Re: syncometer -Likewise the 'syncometer' - no explanation of how it works.

                  ... and have been connected-up with the same rotation.

                  1. swm
                    Boffin

                    Re: syncometer -Likewise the 'syncometer' - no explanation of how it works.

                    In the late 60s I was a student at college and they had a backup generator to keep a drum spinning. Every time the power failed it blew a fuse. I asked if the phase rotation between the mains and the backup generator was the same and got a blank stare until one of the electricians said, "remember the small 3-phase motor we have for checking phase rotation?" They checked and sure enough it was reversed and when the cutover switch was thrown the spinning motor caused enough current to flow that a fuse was blown and the motor happily ran on single phase (it kept spinning anyway).

                    You don't have to be precisely in phase with a cutover switch unless you have two generators simultaneously supplying the same load. During the great northeast blackout I went down to the college generating station and saw three incandescent lights that were used for phasing the college generator to the mains. If the phases had been reversed it would be impossible to extinguish all three lights simultaneously.

      2. Anonymous Coward
        Anonymous Coward

        Re: Parse failure

        UPS - you keep using that TLA, but I do not think it means what you think it means.

  3. M7S

    Every cloud has a silver lining

    In this case it is something newer than the 2e2 debacle to use to berate all the cloud telesales "consultants" who seem to think I want to make myself redundant.

  4. allthecoolshortnamesweretaken
  5. ideapete
    Pint

    HV AC > Sync Via DC Bridge > HV AC = Good SOP , HV AC -> Sync < HV AC = TITSUP every time

  6. Commswonk

    Re: syncometer

    The word you are looking for is synchroscope

    See https://en.wikipedia.org/wiki/Synchroscope

    1. Dan Wilkie

      Re: syncometer

      That's the one, it's been a while and I have man flu ;)

      The ones I used had a circle of LED's, when the right LED was lit up in the right place, you flicked the switch and hopefully everything went fine. But you also had to account for the fact that the switch had a delay, so you had to flick it slightly before. And there may have been a rotary pot involved (it was a while ago and a bit of information overload as it's not my specialty - I was learning it to demonstrate a simulator). 9/10 times I blacked out the entire ship and took out all the infrastructure so I'm an example as to why you need sparkys and not IT guys for this kind of thing ;)

      The guy showing me was an ex-navy chief who nailed it every single time, so I think that speaks volumes ;)

  7. thebigfoot

    Telecity's DRUPS fetish

    Hello,

    * Last year in Italy an old-fart under-rated (water ingress) mains busbar caused an phase-to-phase short on input side of drups set and voila! all of DRUPS output breakers tripped.

    * 1 or 2 years ago in Ireland all of their DRUPS set went to by-pass because of a controller malfunction.

    * This time looks like some stupid contractor caused an phase-to-phase short because of lack of a simple phase sync check.

    N+1 redundancy level and DRUPS not a good combination.

    1. Commswonk

      Re: Telecity's DRUPS fetish

      This time looks like some stupid contractor caused an phase-to-phase short because of lack of a simple phase sync check.

      Take the "r" out of contractor and the sentence still makes perfect sense, and could also be correct.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like