back to article Bank of England to set new standards for when IT goes bad

The Bank of England is expecting financial institutions to be a bit less rubbish when IT goes wrong, it said today. The TSB fiasco that led to customers being unable to access their accounts, followed up by widespread fraud, has caused the BofE's Prudential Regulation Authority (PRA) along with the Financial Conduct Authority …

  1. Anonymous Coward
    Anonymous Coward

    'Nelson worried that "the dominance of just a few providers means that many buyers are not in a strong position to negotiate contract terms with their cloud providers".'

    He's going to be well pleased when the list of cloud suppliers is (ultimately) limited to AWS, Azure and Google, then...

    1. Anonymous Coward
    2. Anonymous Coward
      Anonymous Coward

      Never mind those three. Quake at the providers you don't think of first: IBM, HP and Fujitsu.

      I've worked at 3 banks in the last year, and two of them are big players that you will know. Every one of them has outsourced their IT to IBM and the 'IBM Cloud'. Outsourced as in

      (a) IBM now own all the IT kit. Bank leases it back.

      (b) All our infrastructure is run by IBM

      (c) Some of our IT knowledge was transferred to IBM during ramp up. We fired the rest.

      Modern banks are basically data centres with a banking license ( they are busy closing those expensive 'real' branches as fast as they can.) If IBM flops in the next downturn, the 2007 crash is going to look like nothing, as all the banks that rely on them fail.

      1. Alistair
        Windows

        Uhhhm:

        (c) Some of our IT knowledge was transferred to IBM during ramp up. As soon as IBM could they fired those. We fired the rest.

        Fixed that for ya ....

      2. Anonymous Coward
        Anonymous Coward

        When, not if

        If IBM flops in the next downturn, the 2007 crash is going to look like nothing, as all the banks that rely on them fail.

        I think you meant *WHEN* IBM flops, not IF. Because that's the one thing they're good at.

  2. Kevin Johnston

    and there was me thinking that this was the Bank of England following El Reg's lead and defining grades of crap such as a TSB being a loss of online banking, a Barclays being a total outage requiring rollbacks and so on

  3. amanfromMars 1 Silver badge

    Hmmm?

    He further added that financial institutions should also be regularly testing their incident response and contingency plans, something many IT professionals would regard as Standard Operating Procedure.

    There may be many who, given the right dodgy and self-serving nature of many a financial institution, have made a decision to regularly test monied systems responses and contingency plans with incidents and zeroday vulnerability exploits and phishes.

    The secret sauce then for said financial institutions is to ensure secure slush funded payment/danegeld remuneration to friendly efficient pen-testers so that the monies so earned can be lavishly spent on making fiat currency a much more attractive enabler of novel intellectual property rather than it being squandered to try and destroy it.

    Such though may presently be much too great a quantum leap for all but the greatest of money making machines to make and take, and thus do they condemn themselves to be relentlessly exposed to ever greater losses and the catastrophic leakage of inconvenient sensitive intellectual property which guarantees their self-destruction at the hands of, and in the hearts and minds of, the cynically and designedly dispossessed.

    1. onefang

      Re: Hmmm?

      Are you feeling OK amanfromMars 1? You almost made sense. I was forced to upvote you.

      1. amanfromMars 1 Silver badge

        Re: Hmmm? You almost made sense.

        Howdy, onefang,

        Shall we meet on the fence and agree you very nearly understood all that was shared. It more accurately suggests we are both learning/have both learnt from mistakes made earlier in the past and that is much more encouraging with particular and peculiar regard to the future and what will be shortly further shared for consideration and peer review.

  4. Gene Cash Silver badge

    So basically

    This is the adults having to step in?

    1. HmmmYes

      Re: So basically

      BoE? Adults??

      This is the org that let 80% of the UKs banks blow up.

      1. Doctor Syntax Silver badge

        Re: So basically

        "This is the org that let 80% of the UKs banks blow up."

        By following a definition of inflation laid down by the then PM, former Chancellor. A definition that said increasing house prices didn't count as inflation when setting interest rates because low interst rate buy votes as well as keeping the cost of govt. borrowing down. That definitely wasn't a housing bubble you were betting the bank on. Definitely not, right up to the time it burst.

        1. HmmmYes

          Re: So basically

          Nothing to do with inflation, however it is defined.

          All to do with slack lending, the thing at the core of retail banks.

          1. Doctor Syntax Silver badge

            Re: So basically

            "All to do with slack lending, the thing at the core of retail banks."

            And all to do with low interest rates set by the BoE on the basis of a fallacious inflation figure. Personal debt just kept on climbing because when it's so cheap why not borrow a bit more?

  5. Anonymous Coward
    Anonymous Coward

    Black Swan Attack

    My IT is weak and there is a Black Swan Dancing.

    Will my boss stop the excessive prancing.

    The downtime is here, there is no doubt.

    Oh my, there is even downtime in the cloud.

  6. Anonymous Coward
    Anonymous Coward

    We need a backup system

    I think we need to have some sort of alternative backup system in case of major failure. Maybe some type of metal token or small slips of plastic or paper that you could use in lieu of a debit or credit card if all else failed. Look, we're all super-patriotic these days, so you could even put a picture of the queen on them.

    1. Chris G

      Re: We need a backup system

      Nah! It'll never catch on, loads of metal tokens will ruin the hang of your suit.

      1. Paul Crawford Silver badge

        Re: We need a backup system

        Keep them in a plastic tube then? Should improve the hang of one's trousers.

        Just like the Spinal Tap airport security scene...

  7. Vanir

    Banks too big to fail ...

    but these banks have IT systems that are too big, too complex and too old and getting worse with an increase in the probability of major failure. It's not a question of 'if' now is it?

    Banks are, as Trump would have it (or not), are of national security importance- global too. The IT systems of these national security entities do not appear to be seen in a similar vein.

    It may be time, before a bank merger is allowed to go ahead, the 'proper' authorities look at the state of the parties' IT systems and any plans to integrate those systems over a period of time. I know, I'm dreaming.

    1. Anonymous Coward
      Anonymous Coward

      Re: Banks too big to fail ...

      I think it's pretty clear that the TSB isn't "too big to fail".

    2. Doctor Syntax Silver badge

      Re: Banks too big to fail ...

      "before a bank merger is allowed to go ahead"

      Or demerger as TSB has shown us.

    3. David Beck

      Re: Banks too big to fail ...

      The old system worked. It was the new shiny one that went all to crap.

  8. Ken Moorhouse Silver badge

    While they are at it...

    Algo-trading in my view needs to be heavily "regulated". Stock market meltdown caused by two or more inconsequential "indicators" moving relative to each other in a way that some analyst has concocted as being a "significant" trend. Does anyone know the extent such "algos" lurk in financial trading apps? I suspect many developers are not themselves aware because of complexity (how do you test every input/output?). Then there will be those that, due to a coding error, do something weird for some esoteric input combination.

    I believe that the LSE uses "random" techniques in order to reduce such systems ability to trade pre-emptively e.g., start of auction periods not dictated deterministically by the clock. Maybe having a periodic "lie-in", an hour however being a tad excessive, will assist this disruptive process.

    1. LucreLout

      Re: While they are at it...

      Does anyone know the extent such "algos" lurk in financial trading apps?

      Ken, I can answer this, but please forgive my crude attempt to squeeze what would need to be a small book or long article into a blog post.

      I'm afraid the answer is no. Nobody knows the extent of alogrithmic trading because it is extremely secretive even within an organisation. The department I write algo trading apps for does not share knowledge of them outside the business unit. If your desk knew what my desks algos did, you could write an algo to eat our lunch - make the money in our place. Desks within a bank are in competition with each other to produce revenue and obtain funding from their treasury and var from their RO's.

      Each algo operates in isolation of each other, even within our desk (business unit). I actually do understand the complexity involved in the algorithms my code implements - I won't pretend it's easy, but then we're pretty well compensated for the fact that there's very few people can do what we do.

      I suspect many developers are not themselves aware because of complexity (how do you test every input/output?)

      The testing itself is actually remarkably simple compared to what you'd imagine. Algos ultimately produce a signal which tells the OMS what to do trading wise. The data they take in to generate that isn't a wide data set, but it is fast flowing. You're usually looking at correlated pairs - so if Shell & Tesla move in opposite directions within a short space of time, the behaviour of one factors into your trading in the other - but by far and away most algos trade FX. You're swapping from USD to GBP and back, for example. There's a list of about 10 to 15 common pairs that have a large and fluid market that we use.

      When trading FX you're selling the currency you hold to buy the one you don't. My dollars for your pounds for instance. The data used for that varies but typically will be the tick data for this session and possibly a couple previous, from a number of hotspots (data sources - LSE, NYSE, Simex for example). A tick is one millisecond, so we might have say 10 data items per millisecond to process, but they're all just decimals.

      The current decimal is compared to the others for the tick in question and that is fed into the algo model which decides to either do nothing, or to change the balance we hold in the currency pair - reduce dollars and increase pounds, or increase dollars and reduce pounds.

      The second class of data is the hotspot order book - what orders exist in the market, and where they are positioned in the spread. There are three prices involved - the bid, offer, and mid price. The mid price is halfway between the bid and the offer. Bid is the price you buy at, offer is the lower price you sell at. The difference between them is the spread, which is where market makers profit.

      FX is literally a zero sum game - all that changes is the balance of currencies you hold - for me to hold more dollars, someone has to hold fewer.

      I'm massively simplifying this and trying to use terms at least passingly familiar to most people. Hopefully you can see that the inputs are readily knowable and easy to test in terms of bounds etc. I'm not suggesting mistakes don't happen (google knight capital for a great example), simply offering that the difficulty in testing is lower than that of say a 777 software system, or lane guidance system on a car.

      1. amanfromMars 1 Silver badge

        Re: While they are at it... money grows on trees

        Thanks for that, LucreLout. And when algos crash, where go the losses ..... other than into private pockets/shady and shadowed numbered accounts?

        1. LucreLout

          Re: While they are at it... money grows on trees

          And when algos crash, where go the losses ..... other than into private pockets/shady and shadowed numbered accounts?

          That depends. One of my banks algos could be trading in the opposite direction to 'mine', thus my desk loses and theirs wins - they get a bigger bonus at my expense.

          If the net direction of our algos suffer a trading loss then someone somewhere wins - FX is a zero sum game - my loss is your win and vice versa. It really is that simple.

          Don't confuse algorithmic trading with tax avoidance and don't confuse that with tax evasion - they're three very different things.

      2. Ken Moorhouse Silver badge

        Re: what would need to be a small book or long article

        Come on then.... I'm sure Le Reg would give you sufficient space... and a microphone (if a lecture format were more appropriate). IMHO this is a topic that mot enough people are aware of what is going on behind their backs.

        1. LucreLout

          Re: what would need to be a small book or long article

          IMHO this is a topic that mot enough people are aware of what is going on behind their backs.

          Its not really behind their backs, the only reason the algorithms are secret is that if you knew how our model responded to given market conditions you'd be able to trade against it and eat its lunch.

          I'd love to do a Reg article on this, but I already post way more details about my specialist bit of the bank (tax arbitrage) - any more detail might give away the bank, my identity, and I could quite conceivably get fired for it.

  9. chivo243 Silver badge
    Happy

    the world isn't crashing

    It's just way more reported in the news?

    Yes, yes I used someone's else's logic there

  10. simmondp

    Backup plan??

    I happened to walk into Asda during the Visa chaos, only to be told "it's cash only" - gone of the days of getting out a manual card-swipe machine.

    The reality is that the tech is so complex and interdependent - there is no plan-B when it all goes wrong.

    1. Gareth Davies 2

      Re: Backup plan??

      The manual imprint process still exists.

      1. Phil Kingston

        Re: Backup plan??

        It may exist, but good luck a) finding one b) finding someone in your local Spar who knows how/is willing to use it.

  11. telecine

    See below....

    Cash is king .

  12. anthonyhegedus Silver badge

    It's funny how the scammers get their act together and produce scams to specifically prey on TSB victims quicker than the bank can get their act together.

    1. lglethal Silver badge
      Go

      All a question of motivation...

      It's a question of motivation. Scammers know they need to move quick (well they should have needed to move quick but based on TSB's response they probabaly could have taken their time) to make a profit and so are incentivised to pump out the scams fast.

      TSB know very few customers are going to go through the hassle of changing banks just because of this debacle. And when you know that, then you know there's no need to pay for that expensive overtime to repair the system or invest in things like disaster recovery, roll out testing, rolling back systems, etc. TSB's motivation is to spend as little as possible on things to do with the customers and save as much as possible to go to their bonuses.

      One motivation inspires quick action and the other does not...

  13. Terry P

    I think this is a good thing. Having worked with the BoE in the past for a very long time they are actually excellent IT wise. They analyse the risk of everything, test many technologies and lead with regards to compliance even in areas they don't need to - just to set a standard and be seen to do the right thing. They investigate if they *need* bleeding edge tech or if they are good as is.

    Many of the IT team are long timers or lifers who have turned away significant packages from banks despite having the experience and skill set to meet and exceed the positions offered.

    1. Anonymous Coward
      Anonymous Coward

      BoE IT systems

      Eeerrm remind me - exactly how many direct customers do they serve with their IT systems? Can I get airmiles on their Visa Card? thanks in advance....

      1. Andrew Dancy

        Re: BoE IT systems

        Until a few years ago they actually had several thousand direct customers as staff and pensioners had accounts with the Old Lady. No credit card, but a rather fancy cheque book with sort code 10-00-01 . And back in the good old days they had ridiculously cheap loans and mortgages. Mind you no chance of an overdraft facility - if you went a penny overdrawn you'd get a polite but formal letter telling you not to do it again.

        They also used to have a number of commercial accounts for Government departments such as HMRC

        All scrapped by Mervyn King as part of his drive to turn the bank into a giant economics thinktank and get rid of everyone actually interested in banking. This after a certain Scottish chancellor shafted Eddie, the previous governer, by removing all the regulatory bits and handing it to the idiots in the FSA.

  14. SVV

    Foolproof plan

    All losses incurred by businesses and customers to be deducted from next 5 years bank bonuses, starting at the top and working on down the executive hierarchy, still applicable if executives move to other companies. Restructuring of bonus / salary structure not alllowed during this time. Asset confiscation if attempt to wriggle out of liability any other way.

    Watch the investment in backup and system testing soar.

    1. .@.

      Re: Foolproof plan

      It'd never happen. Too many children of MPs work for banks ...

      And indeed ex-MPs ...

  15. Anonymous Coward
    Anonymous Coward

    Did this get worse when they started "outsourcing"?

    Just wondering.

    If you no longer have staff that know what they are doing. Who sticks their hand up and says "this is a bad idea"?

    1. Phil Kingston

      Re: Did this get worse when they started "outsourcing"?

      I think an El Reg article covering TSB's TITSUP mentioned that there was some hand-raising, but they were over-ruled by those higher-up.

    2. 0laf

      Re: Did this get worse when they started "outsourcing"?

      No one. Which the situation the middle managers want to be in. Top bosses get told about the big savings they are about to get by the implementation of the new outsourced unicorn shit cloud service. The work carried out by some allegedly overqualified 8yr old in a tech sweatshop.

  16. Will Godfrey Silver badge
    Unhappy

    House of cards

    Well, no actually. That has a broad base. This whole mess is an upside-down pyramid , getting rather wobbly on it's tiny contact with the ground.

  17. Anonymous Coward
    Anonymous Coward

    Reality bites

    Cloud is essentially the same IT hosted somewhere else. It will often have very similar properties to IT you would have hosted yourself.

    Except the management perception is that its a sort of magic that makes everything 100% bulletproof reliable forever.

    SaaS providers will use sorcery ensure their systems are far better than yours and that any kind of hiccup with system integrations or data will also be the customers "fault"

    Major assumption that our providers will take a hit if there is ever a problem. reality is that it is your own brand and business that will suffer no matter how much you tell people the root cause was contractual.

    </Rant>

  18. LucreLout

    BoE

    Dear BoE,

    If you're serious about reducing IT outages, then I have a very simple step by step plan that will see off most of them:

    1) Stop outsourcing.

    2) Stop offshoring - those guys are cheap because they are low skilled and inexperienced, not because they're feeling charitable.

    3) Ensure a proper representation of developers and networks staff at senior level - If the CTO hasn't coded in 10 years then he doesn't know what's going on. Sorry, he just doesn't.

    4) Root out bad management by having mandatory 360 reviews, and if the whole team are having issues with the manager, get shot of the bad manager.

    Keep it simple and you'll get results.

  19. StuntMisanthrope

    Government Infrastructure as a Service.

    It's time, it really is, national basic banking for everyone. Income or credits in. Personal tax/NI out. Its a just a data-centre with a Government mandate. Opensource payment API's as the standard. The clever bits can be fought over. #productivity #industry #UKstandard

    1. Doctor Syntax Silver badge

      Re: Government Infrastructure as a Service.

      Right. The team will be with you to start on that as soon as they've finished Universal Credit.

  20. onefang

    Banks keep your money in the cloud? Yet another reason why I don't leave money in the bank. I only have a bank account coz the government insists all payments go through it.

  21. onefang

    Fence it is then. Pleased to meet you, how's the Earth gravity treating you?

    1. amanfromMars 1 Silver badge

      Quantum Progress, onefang ... Where and When This is That and Something Else Entirely Different Too

      :-) When the Status Quo is Moribund and Disease Riddled, Radical Fundamental Revision and Super HyperVision Steps In to Flash Crash SCADA Operations and/or Provide Elite Exclusive Executive Systems Makeovers ..... Remote Anonymous Command and Virtual Autonomous Control Takovers?

  22. Stuart Halliday

    I went to help out my Boss's friends Accountancy business IT.

    The company has 15+ staff and turns around £3+ million . OK. not huge. But then they're handling the accounts of thousands of local businesses.

    I discovered that their 3 year old Backup system, which consisted of a DAT tape getting backed up every night at 2am, was coming up with "Do you want to overwrite this tap?" window and defaulting to no because no one was around!

    The CEO was duly swapping the tape every morning for the last 3 years. But sadly didn't bother to actually check it was working. The set of tapes were full of 3 year old data....

    I wish I had taken a picture of the CEO when he went very pale and then very red and sat down as the enormity of what mess his company and his reputation would have been in if his system had fallen over.

    All I had to do was add /Y to the backup script line. Two ascii characters that would have costed him £3+ million?

    You have to laugh at folks.....

    1. Alan Brown Silver badge

      "I discovered that their 3 year old Backup system, which consisted of a DAT tape getting backed up every night at 2am"

      In the 21st century, what kind of sane person backs up to 4mm tape? Even if it _appears_ to have backed up OK, there's no guarantee that you can read it a year later, or even tomorrow (no read after write and the substrate sometimes goes wonky)

      There's far more wrong with this than the CEO's error and your adding a /y to the script doesn't even begin to cover it (Hint, if the backups are using a script then it's not a backup system, it's a kludge)

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like