back to article 'My entire company is without comms': Gamma's Horizon cloud PBX goes DOWN

Biz broadband comms provider – in name at least today – Gamma is suffering from web wobbles this morning, reportedly across the UK, as customers can't access its Horizon service. The woes for Horizon, which includes fixed and mobile telephony served up via Gamma's cloud PBX system, began just after 09:30 GMT, according to …

  1. jonathan keith

    Still TITSUP* here

    Hello to you all from the balmy Jurassic Coast (Devonian section)

    *Total Inability To Sustain Uninterrupted Phonecalls

  2. BitCoward

    No sign of the promised event horizon here at 14:24 GMT

  3. ForthIsNotDead
    Thumb Down

    Hmmm...

    "El Reg called Gamma and a rep in the marketing department told us it is preparing to release a statement detailing the reason for the outage."

    So... Gamma are clearly not using their own Gamma Cloud PABX to man their phones, then eh?

    I think that tells us all we need to know.

    1. Anonymous Coward
      Anonymous Coward

      Re: Hmmm...

      It’s an amusing point, but I suspect it’s actually a regulatory requirement.

      Imagine if their support team were using their own system in this scenario. It would be even more unpleasant than it looks to have been.

      1. This post has been deleted by its author

      2. TechyLogic

        Re: Hmmm...

        Oh, their resellers use it. It was as embaressing as you imagine.

        I've just got the RFO through. It's so grim im not sending to my customer.

  4. Mikeokurts

    14:43 AND STILL NO SERVICE TO THE ENTIRE COMPANY, Just managed to contact one of our suppliers CEF who also have Gamma, they are out nationwide through the whole group!

  5. Lee D Silver badge

    We're on Gamma, and we're entirely SIP.

    Not seen any problems yet, however.

    P.S. This is why you ALWAYS keep one analogue line...

    1. John Brown (no body) Silver badge

      "P.S. This is why you ALWAYS keep one analogue line..."

      Many years ago, we had a local outage of the phone system around our main office. Every field engineer within a 100 miles was ordered back to base so we could hand over our brick-like analogue mobile phones which BT would then re-route calls to (back then, only field engineers had mobiles, and not even all of them had one. Fecking expensive!!) So, it's not just digital cloudy stuff that fails and it's not new. New, is digital cloudy stuff can have a much more wide ranging impact when some digger driver up-roots a cable or a system update goes TITSUP.

    2. jockmcthingiemibobb

      And zero farks given to any company that doesn't.

    3. Soruk

      We're also on Gamma,and we get a straight SIP feed from them for our number blocks. It's been fine, no hiccups at all.

  6. MrKrotos

    15:00 GMT

    Been told that we should see some stability @ 15:00GMT, we shall see. The map looks nasty! https://downdetector.co.uk/problems/gamma/map

  7. Booty003

    Yep, Gamma down here in the North West. Approx 5 hours.....

  8. Justin 9

    Still no service in Cambridge

    Phones are randomly connecting and disconnecting

    With phone twinning I am able to get calls.

  9. My-Handle

    Our company in Northern Ireland has been down since around 11:00. Still down at 15:20. I can also add that any phone handsets that were reset here lost their config files and apparently have to be sent back to Gamma for a fix.

  10. Commswonk

    "My entire company is without comms."

    Did your mother never tell you never put all your eggs in one basket?

  11. Anonymous Coward
    Anonymous Coward

    Gamma's cloud clusters

    Yet another case of a cloud provider overselling the product. What does Gamma's cloud run on in the backend? While you're at it, please define 'cloud', as it seems to mean something different depending who is telling it.

    It’s still down and more than 30 minutes. 8 care homes without phones. really critical resolved soon. Why can’t we divert phones as promised when entered into contract?!” link

    Because when you signed the contract, you transferred your phones on a VoIP service running on the same cloud service as Gamma, with no provision for falling back to the legacy system.

  12. kingstonkhan

    Phones, where we're going we don't need phones, apparently

    My company is still down, we're based on the Horizon cloud PBX provided to us by Gamma.

    We've only really heard from our internal accounts manager once we started musing about compo in emails to them due to an entire day of lost sales as most of our business is done over the phone.

    The worst part of it is, the system we had in place for disaster recovery, hasn't worked, which has made this all so much worse than it should be.

    1. kingstonkhan

      Re: Phones, where we're going we don't need phones, apparently

      It seems that my comment took much longer to get approved then from when I posted it.

      We currently have gotten around the Horizon issues by activating twinning on a number of users that have mobiles and have forwarded our main hunt group to a BCP hunt group that has these mobile users as members.

      The system didn't come back up at 15:00 as stated in the status updates, probably a much bigger job than Gamma/Horizon thought

      1. diodesign (Written by Reg staff) Silver badge

        Re: "my comment took much longer to get approved"

        New comment posters are manually moderated at first to make sure they're not spammers. That means comments are in a queue waiting for editors with time between articles to run through.

        C.

        1. kingstonkhan

          Re: "my comment took much longer to get approved"

          Makes sense, thanks for clearing that up for me.

      2. John Brown (no body) Silver badge

        Re: Phones, where we're going we don't need phones, apparently

        The system didn't come back up at 15:00 as stated in the status updates, probably a much bigger job than Gamma/Horizon thought"

        Looking at DownDetector, it appears the system (at least for the Leeds area, the only one I checked), was all back up by about 8pm, ie the graph shows reports of downtime at zero from about that time on.

        1. kingstonkhan

          Re: Phones, where we're going we don't need phones, apparently

          We were finally back up at 17:30, when the office closed for the day...

    2. John Brown (no body) Silver badge

      Re: Phones, where we're going we don't need phones, apparently

      "The worst part of it is, the system we had in place for disaster recovery, hasn't worked, which has made this all so much worse than it should be."

      Did the person responsible for the DR plan ever test it?

      1. sanmigueelbeer
        Happy

        Re: Phones, where we're going we don't need phones, apparently

        Did the person responsible for the DR plan ever test it?

        No, but he did get a promotion!

      2. Stevie

        Re: Did the person responsible for the DR plan ever test it?

        "Plan? There ain't no plan"

        Pigstealer

        Mad Max Beyond The Thunderdome

      3. kingstonkhan

        Re: Phones, where we're going we don't need phones, apparently

        Nope, former technician before me was told that it would automatically fail over and that we wouldn't be able to manually invoke a DR state.

        The only silver lining in this, is that I have actually found a way to manually force calls to our main sales number to forward to a group of users that have their mobile numbers twinned with their horizon account.

        However, if the horizon portal is down at the same time, I'll be up a certain creak in a leaky boat.

  13. Paul 87

    Plus side, our partner company has been pretty helpful in ensuring updates on progress (or the lack thereof) are going out.

  14. RodHull
    FAIL

    Fail over fail

    The best bit is you can't get into the promised call redirecting.

    Third party service providers huh.. Next time, sign a contract a little further up the food chain. For us, that'll be in 5 years if the phones come back on.

  15. Anonymous Coward
    Anonymous Coward

    Hmmm

    I hear that the word "Broadcom" is being muttered darkly over at Gamma Towers.

  16. Captain Scarlet Silver badge
    Coat

    That IT Guy

    I wonder if the customers of "That IT Guy" will remember him as that IT Guy for this incident?

  17. Sgt_Oddball

    That explains that..

    Emergency email from a local estate agents explaining the lack of phones.

    Couldn't have happened to a nicer bunch.... (think asking around 15% more than local prices and piling appointments to overlap so it looks like there's more interest in the property)

  18. Dan 55 Silver badge

    The clue is in the name

    If it has 'cloud' in there somewhere it's best to avoid it.

  19. beachmunkey

    Peaceful at least

    Still nothing here - 16.52 GMT and silence!

  20. Alan Brown Silver badge

    This is why....

    Critical service updates should be done in the wee smalls of sunday morning.

    That way you have around 24 hours to fix things before brown smelly stuff REALLY hits rapidly rotating thingy when things don't work.

    Of course PHBs don't like that as it means doubletime (and then some), but it was the normal way of doing things in telcos once upon a time - and it's cheaper than the losses if (when) things go wrong.

  21. SteveTM

    As I've said many many times, Cloud Tel systems are inherently flawed. When it goes down it all goes down and you can forget any portal access / call redirection failover. Its fine under certain conditions (remote workers/DDIs/cheap calls) but for mission critical its really hard to beat copper with solid failover options at many levels.

  22. Paul 87

    Think this has highlighted a serious design flaw in their infrastructure

    Not that the failure happens, I meant we all know that IT can and will fail at the worst moments, but it comes as no surprise that service was finally restored after 5pm. Strongly suspect that the system coming back up took a lot longer than expected because everyone kept trying and trying to use the network, but come 5pm, the volume of calls drops, the number of phone restart attempts stops, and the system gets breathing room to come back up.

  23. zxc123

    All move to the cloud!

  24. blue-eyes

    Flat on its back most of the day

    Our phones finally came back at 4pm. We have kept 1 pstn line for just this situation (and as adsl backup for our leased line) and could divert incoming calls to it. Used our mobiles for all outgoing.

  25. dmacleo

    least I don't have to worry about that...

    course I have to worry about daily stuff here, single digit (F) temps, high winds, 6-12 inches of heavy wet snow dropping tonight, etc so on and so forth. life in rural northeast maine can be....interesting.

    but...as Old Dominion sings....thats a song for another time...

    https://www.youtube.com/watch?v=lZ0QFmzl1kw

    best of luck to everyone dealing with this, hope its fixed soon.

    edit:fixed odd link to video

  26. Anonymous Coward
    Anonymous Coward

    It was only a matter of time

    With the horizon platform down has no one thought about the consequence in such event if a outage beforehand ? I expect the users were attracted by the price per endpoint being attractive at the time of purchase and never thought about the loss of business in the event of an outage . Can anyone afford to lose business I expect not . Long live the onsite PABX !

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon