back to article NAB mainframe turns its TOESUP* after power outage, offline 7 hours

The National Australia Bank has been sharply criticised after a seven-hour outage on Saturday that took down its ATMs, EFTPOS, Internet banking, mobile banking services, and call centre operations. The bank has pointed to a problem in its Melbourne mainframe system as the cause of the outage. By knocking out payment …

  1. Anonymous Coward
    Anonymous Coward

    "main online transaction processing server"

    lol, that twitter guy doesn't have a clue how mainframes work, does he.

    More likely human error where the "cleaner/electrician/technicnian" "unplugged" it in error.

    1. Anonymous Coward
      Anonymous Coward

      Re: "main online transaction processing server"

      It makes perfect sense if both were running off the same power supply and they didn't have a backup power supply. Before they brought up the mainframe they'd need to shut down the ancillary servers and flush their queues. Bringing the whole network back online will take time - which is why you have backup power, and regularly test it.

      The more important question is how does El Reg determine if a system goes TOESUP or TITSUP? I blame the GDPR.

      1. Long John Brass

        Re: "main online transaction processing server"

        I'll bet you my non-existent Christmas bonus that someone complained that TITSUP was sexist/offensive/triggering

        1. Anonymous Coward
          Anonymous Coward

          Re: "main online transaction processing server"

          You are, unfortunately, probably right. And I don't suppose switching the gender to the far more traditional "COCKUP" would be acceptable either. As a wise man said, things change, things get worse.

        2. rskurat

          Re: "main online transaction processing server"

          I'm male and 45kg overweight, and I can assure you I do in fact have tits. I was not offended.

      2. mc nobby

        Re: "main online transaction processing server"

        Yeah, there really isn't any excuse for a Core system to have no power. There should be layers of redundant power, most data centres have DRUPS these day's, and if that fails they have Diesel generators, which you hope fire up because you test them regularly. So if all the systems have power, you would have to assume it's some other System Cockup that's been the issue.

        1. moooooooo

          Re: "main online transaction processing server"

          NAB has 6 diesel generators at the Knox data centre IIRC. So IMHO it wasn't a power outage at the data centre. Unless they've moved the mainframe to the new (to me) data centre which was commissioned yonks after i left. Rumour has it that it was related to switches dying but i dunno.

  2. Pomgolian
    FAIL

    Also Broke BNZ in New Zealand

    New Zealand's BNZ are owned by NAB, and there was much wailing and nashing of teeth over the ditch for a few hours when nobody could pay for a latte or petrol etc. Cashpoints and cards were out.

    1. Yes Me Silver badge
      Flame

      Re: Also Broke BNZ in New Zealand

      Must be a crap system design with no thought of resilience.

      1. Griffo

        Re: Also Broke BNZ in New Zealand

        Actually knowing a little about the NAB's datacenters, power setup and mainframes, this is not an everyday failure. They have several layers of power redundancy at their main Knox DC and their new secondary facility is state of the art.

        1. Anonymous Coward
          Anonymous Coward

          Graduates

          If by "state of the art" you mean their Deer Park facility, let's not forget is was built by IBM. Which don't exactly have a stellar reputation for outage management or reliability in Australia lately.

          1. Anonymous Coward
            Anonymous Coward

            Re: Graduates

            Wasn't it built by builders?

  3. GrumpyKiwi

    Cost us about AUS$2000 in lost sales - based on Saturday averages over the past couple of months - and (more importantly) it cost me my nice relaxing Saturday.

    Be interesting to see how much compensation they offer.

    1. Long John Brass

      Be interesting to see how much compensation they offer.

      Somewhere between bugger all and fuck all would be my guess

      1. Phil Kingston

        Sadly, I fear you may be right.

        And I'd imagine it'll be a while before they get round to a heavily-lawyered response that'll need signing to confirm the account owner has agreed to their offer of nowt and have no intention of sueing them (now or ever) just in order to maintain an account with them..

      2. GrumpyKiwi

        I'll post it when I find out.

        1. GrumpyKiwi

          Our NAB Customer Manager has told us we'll be compensated for lost margins.

          Which isn't bad according to the CFO, so I guess we'll live with that. Call it a 'Black Swan' once-in-fifteen years event and ignore the head of Retail who wants some kind of backup EFTPOS solution implemented.

          1. This post has been deleted by its author

      3. Denarius
        Meh

        compensation

        Yeserday thought I heard a NAB spokes weasel on radio (yes, big radio) explaining the assessment that will be made. Something along the lines of requiring documentation for each "lost" sale. I also note the other routine Oz disaster, NBN have cut off local businesses when connecting them to the NBN cables. Phone dead for 4 days now as copper cut somewhere. No network as usual. Who would be a small businessperson with all the predators charging?

        About time a few CEOs and boards had their assets stripped under Proceeds of Crime legislation for making false representations, thus winding up doing a few years hard labour instead of a shell company.

  4. Phil Kingston

    Sadly, these outages seem to be an annual event. If not NAB, then one of the others. Luckily I'm not a business-owner. But as a consumer I have a second, pre-paid card with me at all times for just such an eventuality. I suppose cash would be a better backup, but I'd just blow that on beer and crisps.

  5. Tim99 Silver badge
    Coat

    So

    Nothing to do with underinvestment, offshoring, and not having any qualified experienced local support staff then?

    Mine has a copy of the children's illustrated pocket-guide to "Your Business is IT - Not Traditional Banking" in the pocket >>===>

    1. Anonymous Coward
      Anonymous Coward

      Re: So

      Much of the IT involved is mangled by a certain other large corporation, frequently mentioned here on El Reg for their apparently never-ending slide to oblivion. 3 letters, 1 guess.

      NAB is actively trying to get away from them as fast as is possible - which may be near impossible for mainframes.

      AC because... also, 1 guess.

      1. This post has been deleted by its author

      2. sanmigueelbeer

        Re: So

        Much of the IT involved is mangled by a certain other large corporation, frequently mentioned here on El Reg for their apparently never-ending slide to oblivion. 3 letters

        I was about to say that only IBM could make a total cluster-f*ck like this. And I was just making an un-educated guess. I wasn't expecting it was.

        Our NAB Customer Manager has told us we'll be compensated for lost margins.

        No they won't.

  6. chrisw67

    Compensate their customers.... and who else?

    I do not bank with NAB but two of three transactions I attempted on Saturday were nonetheless forced to cash-only because the merchant was with NAB. I went to an ATM and withdrew cash. Not a big deal for me but I imagine a lot of people did this generating an abnormal run on ATMs. Restocking cash in unusually depleted ATMs will cost the ATM operators (other banks and private enterprises) something that they will not get back from NAB.

    NAB best not charge businesses for depositing the cash they took on Saturday.

    1. TReko

      Re: Compensate their customers.... and who else?

      All this makes you wonder about the government and bank pushing towards a "cashless society".

      Cash is a great backup for when things go TITSUP

      1. Denarius

        Re: Compensate their customers.... and who else?

        @TReko

        agreed. Very common here in the boonies, a mere 80 km from the nations crapital. Much worse further out

  7. Tech Brains

    What aren’t they saying?

    I’m pretty sure there’s more to the story. Mainframe going down is a pretty rare occurrence, even today. A data center without redundant power? Where? That’s extremely rare—even in locations where the equipment is being housed internally.

    Just seems fishy to me.

    A month ago Canadian Stock Exchange went down. Screwed trading badly. They’re still trying to recover. It was a big deal. Only one to report it was CNBC. The outage was due to a “storage failure”. Rumor has it that the storage was Pure. Not a single word about this from the Register.

    My point is, more info. Cover it in depth not just high level overviews.

    1. Brett Weaver

      Re: What aren’t they saying?

      To be fair you are reading The Register..

      They can only afford 1.5 full time journalists because you readers wont pay them a subscription! They are forced to just regurgitate press releases for most stories.

      Thank God for the BOFH...

      It would be nice to once in a while find out what really went on and who the culprits were..

  8. This post has been deleted by its author

  9. Griffo

    UPS upgrades

    Inside word is they were undertaking a UPS upgrade and it went wrong.. apparently it caused a set of rolling failures which took some time to untangle.

    1. DoubleBarrelSnotGun

      Re: UPS upgrades

      Not new at all, when I worked for IBM in the min 2000's we had two outages at it's major Sydney DC that impacted at least 2 airlines, multiple banks and many, many other businesses, fortunately around midnight or so both times.

      The first involved some wanker driving his car into the local power substation, which caused the DC to switch to UPS, whose batteries hadn't been checked for far too long. Causing the entire DC to go dark before the generator could kick in.

      The second was when testing the newly installed batteries, some other wanker had forgotten to turn the diesel supply back on to the generator with expected results.

      IBM manglement has not had a clue for years, ignoring tech advice from tech staff is the start of it...

      The more things change the more they stay the same.

    2. sanmigueelbeer
      Happy

      Re: UPS upgrades

      Inside word is they were undertaking a UPS upgrade and it went wrong

      If this is true and IBM screwed this up, IBM will be in for a huge penalty fee.

      How much is a used blue helicopter worth? Slightly used.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like