back to article Unexpected MySQL database meltdown fingered in GitHub's 24-hour website wobble

Programmers, your snow day is well and truly over: GitHub's website has finally cleared its 24-hour outage, and reckons everything is operating normally again. Last year, CEO Chris Wanstrath said the company was shooting for zero downtime, or at least five-nines of uptime. Well, that comes out at about five minutes of non- …

  1. Anonymous Coward
    Anonymous Coward

    "only impacted website metadata stored in our MySQL databases"

    ... which of course is the only valuable part of Github.

    The actual repository data is completely mirrored in every single git checkout that you have.

  2. EvaQ
    Paris Hilton

    Solution by Microsoft

    Problems with MySQL? The new owner Microsoft has the solution ready: SQL Server!

    1. werdsmith Silver badge

      Re: Solution by Microsoft

      Will be an interesting migration project, until then MS will be running using Oracle software.

      Hint: if you think you need MySQL consider Maria first, avoid getting larried.

      1. lsces

        Re: Solution by Microsoft

        Avoid problems altogether and switch to Firebird ...

        1. werdsmith Silver badge

          Re: Solution by Microsoft

          Firebird looks great and something I need to get under my hat.

          Thanks for the heads up.

    2. This post has been deleted by its author

  3. Charlie Clark Silver badge

    MySQL on a production system?

    Insert sneering remarks here…

    Elsewhere it looks like Atlassian had a couple of hiccups yesterday as well. At least the post-commit hooks didn't seem to kick off Pipelines.

  4. jake Silver badge

    OH KNOES!

    THEY SAID FINGERED! The politically correct crowd will be along shortly to wring their hands & whine at ElReg ... What ever shall we do!?‽‽

  5. David Harper 1

    No MySQL replication slaves for failover?

    Any competent DBA will tell you that you should be running multiple replication slaves of any mission-critical MySQL database server, and they should be using different back-end storage systems than the master. These replication slaves can provide read-only copies for load-balancing, and can also be promoted to become a read/write master in the event that the master suffers a hardware failure.

    Someone at GitHub/Microsoft evidently didn't get beyond chapter 1 of the "How to be a production MySQL DBA" notes.

    1. Charlie Clark Silver badge

      Re: No MySQL replication slaves for failover?

      GitHub is a typical Silicon Valley "success". Written quickly using Ruby on Rails, became popular and attracted VC funding and subsequently sold to Microsoft. Do you think there was ever a DBA involved?

    2. phuzz Silver badge
      Gimp

      Re: No MySQL replication slaves for failover?

      Must be fun to be Microsoft, to get the blame at a company they only bought a couple of months ago, for a problem that was probably set up to fail by decisions made years ago...

  6. batfastad

    To be fair they have a very talented and well-respected MySQL team at Github.

    I would reckon the issue possibly with their Orchestrator open source tool https://githubengineering.com/mysql-high-availability-at-github/ https://github.com/github/orchestrator

  7. avakum.zahov

    Time for NewSQL?

    How about using a truly distributed, always available, active-active SQL DB like CockroachDB (if one want to stay with open source)? Come on guys, let's keep the innovation spirit alive.

  8. Mike 16

    Five Nines

    OT, but that reminded me of the "pep talk" we recently acqui-hired remnants of a once-promising startup were given on arrival at the (larger head-count than the town I grew up in) megacorp. The speaker was the (insert plausible title translating to "high muckety-muck") of global marketing, and he reassured us that the emphasis will still be on commitment to quality and reliability, promising we would deliver "nine fives". A fellow newly-borged coworker and I looked at each other, agreeing that this lot could probably achieve it.

  9. Claptrap314 Silver badge

    Microcurse

    Microsoft announces acquisition of Skype--Skype goes down same day.

    Microsoft announces acquisition of Gituhub--Github goes down shortly.

    Coincidence? I. Think. NOT!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like