Sign in / up

back to article Cisco Webex meltdown caused by script that nuked its host VMs

Cisco has confessed that the cause of the mega Webex outage last week – which it is still trying to clean up – was an automated script "which deleted the virtual machines hosting the service". A chunk of Webex was KO'd last week, including Teams, Calling, Meetings, Control Hub, Hybrid Services and more, as we reported at the …

COMMENTS

Post your comment
House rules Send corrections
Add to 'My topics' unstar *
  1. Anonymous Coward
    Trollface

    BofH

    > The service interruption was the result of an automated script running on our Webex Teams platform which deleted the virtual machines hosting the service. This was a process issue, not a technical issue.

    If False == PayrollLookup("BoFH") Then RunTidyUp(confirm=True)

    12 0 Reply
    1. Flakk
      Reply Icon
      Pint

      Re: BofH

      Heh. Reminds me of a classic BOFH joke:

      > Yes means No, No means Yes. Delete all VMs? [n]

      4 0 Reply
  2. Anonymous Coward
    Anonymous Coward

    'This was a process issue, not a technical issue'

    Oh really, WTF??? Looks like automation failure / technical issue to me.

    But hey, re-purpose language / leverage legalese to falsify Uptime stats.

    2 0 Reply
    1. Anonymous Coward
      Reply Icon
      Anonymous Coward

      Re: 'This was a process issue, not a technical issue'

      A computer will do what you tell it to do, not what you would like it to do. There was no technical failure, automation worked as designed.

      3 0 Reply
    2. Trixr
      Reply Icon

      Re: 'This was a process issue, not a technical issue'

      Nah, there is a difference in any failure analysis between a component failure and "human factors" (aka C*CKUP).

      Sounds like the script worked perfectly. How it was *targeted* is something else.

      0 0 Reply
  3. Roger Lipscombe
    FAIL

    This was a process issue

    I expect to see this in "Who, Me?" in a couple of months...

    9 0 Reply
  4. Anonymous Coward
    Anonymous Coward

    The Cloud...

    Other peoples computers....

    2 2 Reply
    1. Anonymous Coward
      Reply Icon
      Anonymous Coward

      Re: The Cloud...

      Buying services....hopefully you get something...

      Or a refund

      Or move to a competitor

      Or decide you have the expertise to provide the service in-house

      Just like so many other options in IT....and life...

      0 0 Reply
  5. Jay Lenovo
    Terminator

    It was the humans, the humans did this

    New processes are in place to stop bad processes.

    CiscoWebex 9000 does not make mistakes.

    5 0 Reply
  6. JMaldonado

    Borg cant assimilate script...

    0 0 Reply
  7. sanmigueelbeer
    Unhappy

    WTF have you done???

    Outage was caused by a "process failure". Translation: Someone didn't follow the procedure.

    So someone knocked up a script which went about deleting the primary AND secondary VMs. After the initial WTF (and subsequent WTF-have-you done), it took another 10 minutes to stop the script's "rampage".

    I hope the guy is still alive.

    0 0 Reply
  8. aaaaaaaaaaaaaaaa

    AI????? hahahaha...

    1 0 Reply
  9. cs9

    WebEx? More like Web Ex

    This is one of those times you find out if an organization is truly committed to the blameless postmortem.

    0 0 Reply
  10. Erik4872

    Blameless post-mortems?

    I _highly_ doubt there will be "blameless post-mortems" in this case like DevOps calls for. Too bad for the person who actually triggered it!

    0 0 Reply
  11. TheyHaveBuiltAConservatory4u

    At least my music wasn't deleted

    All things considered, I am grateful my music library wasn't deleted. I can live without a few conferences.

    0 0 Reply

POST COMMENT House rules

Not a member of The Register? Create a new account here.

Forgotten password ?

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like