Oh no, RBS has gone titsup again... but is it JUST BAD LUCK? And here we go again, another IT systems failure at RBS. RBS appears to have been having a remarkable run of high-profile core-system failures, but I suspect that it has been rather unlucky or at least everyone else has been lucky. Ross McEwan, the new chief executive of RBS admitted to us in a canned statement that "decades …
Those people at the top, They are really in touch with their organisations. They think they are worth the millions of pounds that have spent on them to save money, yet they cannot seem to invest some of the money correctly into core systems that keep the business running. It seems like the technical people that develop and keep the systems running are the real masters of the universe, not the bankers.
It seems like the technical people that develop and keep the systems running are the real masters of the universe, not the bankers.
Yup, but the technical people move on to another job when they find that there is no career progression or salary increases in their current role. Either that or they get made redundant when their role is outsourced. The turnover in IT staff is a huge problem, as we're seen as a commodity or component that can be simply be replaced with another programmer, sysadmin or DBA.
For instance, I had a role where I had architected and lead the implementation on a quite sophisticated system. This took three years, during which I got no salary increase. I then lead the support and enhancement of this core system, as well as implementing a few others on the underlying framework we'd written, for another two years. This included a painful merger, when we were verbally promised improved salaries if we stayed loyal. Then we were finally told there would be no salary increases.
So with a salary eroded considerably by inflation, I quit. The DBA then quit. The other non-contract programmer then quit. It took them months to replace me, as they actually lowered the salary they were offering for my replacement. The person they finally got didn't understand the core system and completely screwed it up.
> Those people at the top, They are really in touch with their organisations. They
> think they are worth the millions of pounds that have spent on them to save money
It doesn't matter what they are worth. It matters what they can get paid.
Sorry, but as I get older I realize that I'd rather be wrong and rich, than right and poor.
who is brave enough to argue for budget to go back and fix those things which aren’t broken
As one business mentor told me, "nobody gets promoted for preventing screwing-up. Nobody gets promoted for taking preventative actions"
It's no joke. In my professional role, I work with C-Level execs in banking, education, manufacturing and retail. All but a very few have even so much as the faintest clue about the department or organisation they are running and make frighteningly poor decisions having been fed a line of spurious ill-informed nonsense. It's not just only a case of 'the Emperors New Clothes' - his courtiers are also parading around with their backsides hanging out as well.
There is an insidious level of greed and short-term opportunism in UK 'Big Business' today that leads to short-term penny pinching with no view to the long-term good of the whole business - all driven by a bonus culture and a complete lack of accountability. If you off-shore or out-source your IT systems, you deserve everything you get - unfortunately, you'll probably be long gone by the time everything comes tumbling down and no-one will be able to pick up the mess you left behind.
It's no joke. In my professional role, I work with C-Level execs in banking, education, manufacturing and retail. All but a very few have even so much as the faintest clue about the department or organisation they are running and make frighteningly poor decisions having been fed a line of spurious ill-informed nonsense. It's not just only a case of 'the Emperors New Clothes' - his courtiers are also parading around with their backsides hanging out as well.
Nowadays people don't become C-level executives by business skill, they do it by holding a black belt in corporate politics, period.
And the courtiers have figured out that they can garner power for themselves by controlling the information they feed their bosses.
In the end, in both the US and the UK, the corporate systems are broken because management no longer has a vested interest in the company's success. The rich get richer and everyone else gets screwed.
" It seems like the technical people that develop and keep the systems running are the real masters of the universe, not the bankers."
But the attitude is IT is a cost centre, not a profit centre. Every penny spent on IT is wasted but spend millions on X manager or Y sales persion or Z broker, why look at how they are cutting costs/selling stuff/????
Every bank and insurance firm in the land is in the same postion.
> But the attitude is IT is a cost centre, not a profit centre.
It is a cost, not a revenue centre, but there are so many over-cost projects which don't deliver savings and/or are not at all robust.
The general trend in s/w development appears to be "larger, more bloated and buggy" so rewriting is an unattractive option.
'The general trend in s/w development appears to be "larger, more bloated and buggy" so rewriting is an unattractive option.'
Do one job, and do it well. MS has seen to the death of the Unix philosophy.
Also, if these companies ran on F/OSS the code could still be maintained/ported/documented. The only sauce they need keep secret is the configuration/business rules that the code processes. £10 + £1 is still £11 whatever way you cut it.
"Also, if these companies ran on F/OSS the code could still be maintained/ported/documented"
That's not the issue here, the code is maintained, it's just that it's been maintained to a point where it very rarely needs any active maintenance. Also, it would have been impossible for these systems, which date back 30+ years to have been made on FOSS as there just wasn't any then.
These days, where is the FOSS CICS or the FOSS database to rival DB2 or Oracle? They just don't exist. They also certainly don't exist with anything like the track record to run core banking on. RBS do run large amounts of Linux on Z and Intel but these are not core systems. Also the requirement for support means that they can't just go about changing the code and a bank doesn't do OS development - it's way out of core business!
The OS that run the mainframes used to be distributed as source (OS/360,OS/VS2,MVS), but customers didn’t want it, and bits of the IPL code that boots it is older than some pensioners. The code is not the issue, it’s the people. Fact is a 55 Systems Programmer can’t be bought with a promise of promotion; its cash or dash. A 40 C-something has no real understanding of what that guy does and doesn’t believe him anyway.
We’re about to get one of those lessons in what a promise is really worth to a bright 20-year on (UK) minimum wage looking for a route overseas.. meanwhile HSBC & Barclays will pay what they need, and we’ll never hear about it.
I applied for a job with a large retail group in the 80s. I found out that part of the job description was to transfer the entire stock database system to new hardware with a complete rewrite using a language as yet unspecified. When asked what the souce was they replied 'assembler' and I replied 'byeeee"
"When asked what the souce was they replied 'assembler' and I replied 'byeeee""
I think you might have missed an opportunity there... Rock solid spec (ie: the code), writing code in the language of your choice... I think your fear might have let you down badly there. ;)
I have found reading assembler written for "big" machines (eg: VAX, S/360 etc) by old school programmers (you know, the guys who inclued verbose comments describing the pseudo code for the program) to be a lot easier than reverse engineering a system written in a mixture of VB6, VBA, SQL, C#, C++ and Java for example...
When Usenet was new, I saw a couple of posts discussing a kernel bug. The discoverer posted the method of replicating the fault - instructions on which programs to run and what options to use. This was followed by the warning that doing this would reliably kill your machine (A Sun platform IIRC).
A few posts later, someone replied "Oh yes, it does - doesn't it".
Could it be that someone within RBS had identified the cause of (one of their many) cockups, told his/her/its colleagues - one of whom then replicated the conditions ...
I had a colleague trying to extract data from a live account system. He used somewhat unusual methods which effectively scraped the data he wanted on an individual account basis. The side effect of this was only noticed when someone a few desks down from me complained that they couldn't access the accounting system (which has about 20000 users across many centres/countries). I asked him to stop his routine and the system started responding again. He was convinced his code was not to blame, so we waited half an hour, noting no issue with the system, then ran his code again. This time we spent a few minutes on the balcony overlooking the main sales floor. Everyone's system was not responding. He finally conceded that it might be his fault. He stopped it, and it all sprang back into life. He has since moved on to be our main solutions specialist. In all fairness, he was grappling with an unfamiliar system designed by committee and patched too many times to be efficient.
Maybe somebody in RBS is trying something equally silly.
> He was convinced his code was not to blame
And that's the problem. While his code worked, it sounds like it had many unforeseen (and on a production system: unforgivably untested for) side-effects: possibly including high I-O usage, unintended database locks, de-cacheing, or futzing (technical term) with the middleware - among other possibilities.
Too many outfits I've worked at regard "testing" as merely checking the known input produces the expected output. They don't bother checking boundaries, error handling, scalability or resource-hogging. Since a lot of these other tests would require having access to a complete replica of the production system - with a simulation of all the users and their workloads - most companies can neither afford, nor organise such a massive testing operation.
"Too many outfits I've worked at regard "testing" as merely checking the known input produces the expected output. They don't bother checking boundaries, error handling, scalability or resource-hogging."
Indeed. During a SAP test at my old employer I was told off for going off the script and trying to break it by putting in stupid data (letters in value fields etc), or forcing it to do something it shouldn't. "We don't have time to test that" being the answer. Oddly enough, launch was an unmitigated disaster as the users all tried to circumvent the controls (successfully) or put in data in the wrong fields breaking reports and searches. Ho-hum.
This post has been deleted by its author
While I do agree with the contents of the article... just look at things from the perspective of the bank - as a money raking machine - and not from an IT best practice perspective.
The mistake is to think retail banks consider their consumer facing systems to be "mission critical"....
Retail banks funnel money in to allow their investment bank buddies to play the stock market (and these systems get all the budget they need)
Providing services to the customers comes second fiddle at best...
So, If it ain't broke don't fix it... and keep it running until it breaks.
If you're lucky you can run for years without any major additional investment in the system
And when it breaks (because it will) then just put a contrite face on - fire a scapegoat CTO - give a few quids of compensation money to your punters - and invest (minimally) into a new system (or a patch.
This "mess" will cost you much less than properly managing and upgrading your legacy systems....and will be forgotten by the punters in a few weeks....
Verdad! They also take advantage of people being too busy or too lazy to switch banks. They always bank on angry client's anger dissipating quickly. In short they have us by the cojones. And you're right about IB money, customers exist to fund trades, because those fund real bonuses!
...or better still Outsource the management of your systems, ideally to the lowest bidder, then you can pretend all is well while they (the Outsourcer) does minimal management (no updates, no patches in case such changes introduce a service interruption) and produce lovely charts each week/month showing how SLAs have been maintained. Eventually the lack of investment will catch up with you and there will be a costly/ embarrassing outage but you will either a) be long gone with a hefty wedge in your back pocket or b) be newly into your role therefore not responsible for the mess. In either case, you can fall out with the Outsourcer and both part company at the end of your 3/4/5 year O/S deal and enjoy the thrills of retrospectively updating systems that are back-level by 5-10 years
...that there is a naming problem here. When Ross says "systems", he's not talking about the tin on the floor, or in many cases the middleware or OS. All of these things are easy to replace.
What he's talking about is the "business system", which is the bespoke software and operational procedures. Much of this code is touched as little as possible.
I think you have picked up on this, but most of the media have not. They are still thinking that the banks have mainframes from before the millennium.
I can promise you this is not the case. Their machine rooms will be mostly filled by hardware less than 5 years old, with very regular maintenance and patching processes. If they did not, the FCA (the regulator that replaced the FSA) have the ability to suspend the bank's operating license. Believe me, this provides huge motivation to the bank's management to make sure maintenance happens.
I got caught up with this at my last stint in one of the large banks about 6 years ago. After an internal audit to check compliance, it was decided that if there had been an external audit of the maintenance and patching regime, the bank would have been severely embarrassed by the results. Suddenly, getting the most recent OS patches and firmware updates onto as many machines as possible became the highest non-operational priority within the support organisations, and at one point, almost all of the support personnel within the part I was working in were involved in one way or another.
In this instance with RBS, I heard on the grapevine, very indirectly, that the problem was with a piece of privileged third party monitoring software which interfered with the sysplex locking structures, causing database access to fail across the whole sysplex. It appears that they effectively had to restart part or all of the sysplex to restore the function. Whether they had to roll back the database(s) and re-apply transactions I do not know. Speculation is that this was a known but obscure fault.
That is not entirely true. While it is completely correct to highlight the business system as the main problem, a lot of banks ARE running legacy systems on ancient hardware. Various banking regulations actually prevent older data/processes from being updated. If you look at ebay you can see certain types of old hardware get pounced on for spare parts to keep these running.
It varies from bank to bank.
After the Millennium, Barclays (my main experience) spent significant amounts of money and a lot of effort consolidating as many systems as they could onto a maintainable platform (they called it project "Stretch"). The aim was to eliminate as many of the old systems as they could.
This spanned the Mainframe, and most of the UNIX platforms they ran.
The main aim was to get all the systems running on a smaller number of larger, more cost-efficient systems that were virtualised/shared and could be bought in volume, and managed more easily (sound familiar - they were well ahead of the curve, and doing this before Hyper-V and VMWare were invented or regularly deployed). But a side effect was that the systems could be maintained more easily, and could be subject to regular OS updates,
They did not completely succeed. Sometimes, the specifics of code that cold not run on more modern versions of the respective OS (I remember Sybase and Informix causing some problems). Sometimes they required specific hardware that no longer worked on modern systems (hardware encryption cards were the most serious problem IIRC). But where this happened, the applications were flagged as in serious need of replacement, and the internal businesses warned that they were in significant danger of losing their service and breaching audit requirements.
But the penetration was more than 80% when I left that project in 2005, and got better after that time, I'm led to believe.
In general, the only stuff that really got left behind was the international banking systems for minor countries (like African countries), where upgrading the remote systems was complex and difficult because of their location, so the local systems were kept largely in line and on the same hardware. But even this was being addressed. In all of these cases, there was an explicit risk evaluation done by the internal auditors, so the respective internal business managers were well aware of the risks they were taking.
So legacy systems were the exception rather than the rule, and were known about. I'm sure that most of the other large banks in the UK are the same. Certainly not what is being implied by some commentators.
"I can promise you this is not the case. Their machine rooms will be mostly filled by hardware less than 5 years old, with very regular maintenance and patching processes. If they did not, the FCA (the regulator that replaced the FSA) have the ability to suspend the bank's operating license. Believe me, this provides huge motivation to the bank's management to make sure maintenance happens."
Doesn't work like that. They have to have support for all equipment not a certain age limit. If a vendor agrees to support it then they can run kit into the ground which is why a lot of financial's are still chuking big bucks at MS to keep Win2K and Xp boxes going
I'll concede that. But if an old system is still under support, and the patches are the latest that vendor/support agent have, it meets the audit requirement. But you would hope that a system that is so old is (a) backed up by a sufficient spares pool, and (b) had most of the bugs rattled out of it over time.
But you're not going to find an IBM 3090 still running anywhere in a UK bank, because it is has been easier and cheaper to migrate onto a more recent mainframe, even if you have to run the application in some compatibility mode because you can lo longer compile it.
You will note that I used "mostly", and I did not say that the FCA mandated that the systems were younger. It's just that most banks can easily spot that maintenance is an FCA requirement, and the cost of maintaining older kit is expensive. Most of them do actually replace kit when the economics are right, especially with the vendors so keen on selling new kit. What the FCA do is make sure that they have a maintenance strategy, and that it is being followed.
The banks can no longer conveniently forget that piece of critical kit sitting in the bottom of the rack in one of their old machine rooms, because they can quite realistically be forced to stop trading as a bank if the regulator believes they are seriously neglecting their responsibilities. This has sunk into the bank management. Being the chief executive of a not-bank when the licenses were suspended would seriously damage the chances of another high-profile job.
I'm quite sure that this RBS incident will trigger an FCA investigation at some level, and there will be questions asked about the maintenance state of the systems involved.
I hear the statement above re IT being a cost.
Whilst its an interesting point , it's also a point worth referring to when you think about old IT and new / newer IT infra.
Many are moving to a... use my kit , pay me money ..model , ie a charge based model.
In which case IT for some is becoming a part of the charge base , if you make Money we all make money and not the poor relation of old.
Worth noting , when anyone talks about IT and old kit... In future .. If its old its probably costing you a fair wedge (remember support costs on old kit , people) if that wedge is a cost , then your "wheels" and support within the org is poor, you're onto a loser and as ,quite rightly, the storage bod points out , it's only a matter of time.. Before those "wheels" come adrift .
>Many are moving to a... use my kit , pay me money ..model , ie a charge based model.
>In which case IT for some is becoming a part of the charge base , if you make Money we all make money and not the poor relation of old.
What could possibly go wrong with moving all of your mission critical business systems into the cloud? I am sure your provider has your interests at heart. The NSA snooping is just bonus.
Isn't this a wider problem? Who writes code with the expectation that it will need to be replaced or migrated in ten or twenty years in the future? Look at all the recent grief given MS for daring to say that Windows XP will meet it's "Sell by" date next year - and really it should have been replaced several years ago.
But computer programming is still in its infancy compared to Banking and we're still struggling to figure the whole thing out - sure, there are computer languages that have some modicum of portability and long term evolution but we've mostly thrown them out because the new kids on the block are sexier and offer mood of the moment features - who bothers to learn COBOL, FORTRAN, or ADA any more?
Old stuff - only old farts write in that stuff - we use VB ... no, wait lets code the new ATM with Ruby or Java ... and so down the road we discover that both our new fancy languages have more holes than an aerosol sandwich. I'm not arguing for a wholesale return to writing banking applications in COBOL (assuming you're still with me and haven't down-voted this and moved on) but simply saying that we need to learn from the history of application writing - COBOL et al have legions of issues but the applications worked and were/are supportable for a long time ... do you really want to be debugging Java code that's twenty years old?
But nobody ever thinks about the future when they write the code these days - just get it out of the door, collect the bonus and move on.
"who bothers to learn COBOL, FORTRAN, or ADA any more?"
I know COBOL. It was good at what it did. Would I want to write a GUI app in it? Hell no. Would I prefer it to the likes of Java for tearing through masses of formatted files/records? Hell yes.
"do you really want to be debugging Java code that's twenty years old?"
Even debugging Java code that's two years old makes me cry.
Software Engineers who are interested in making their code work with the above - or even fixing it. You can write unit tests and- fuck me - good comments in all the above. Just because a language is old doesn't make it immune from good software practices. But I bet the budget and management probably do.
"who bothers to learn COBOL, FORTRAN, or ADA any more?"
Ada flies aircraft. No Ada, no fly (there are obscure alternatives, arguably safer and quite possibly lower cost ones too, certainly flight proven over many years, but because they are obscure and contractors therefore cannot be instantly recruited from the jobcentres, the High Ups prefer Ada).
The problem is that COBOL / FORTRAN are not *cool* languages any more. But they are absolutely suited to the purpose they were designed for: Processing transactions - ie shunting around text and numbers (with a little bit of simple maths) from one place to another. Because these systems are very stable - ie: the language itself is stable & subject to few changes and the business processes are stable, few changes are required, so these languages (COBOL etc) appear to be in decline because there is relatively little continued development effort required. In reality many, many more transactions are handled by this old code thanks to the increased capabilities of modern HW.
So now we have a dearth of COBOL coders and a glut of Java/VBA/C/C++/C#/Javascript coders - so what happens, these (stable and working) systems get re-worked or interfaced to by Java/VBA/C/C++/C#/Javascript code and these Object Oriented languages are not optimised for this sort of simple transactional workload.
Dependencies on 3rd part frameworks, overly complex syntax, deployment onto cheap wintel servers add unnecessary complexity and fragility. Simple truth is that pre-Windows GUI (and now Web UI) these systems ran on simple Green Screen Dumb terminals (remember them?) they were more efficient, the staff were more efficient and the maintenance of these systems was simpler, more effective and less costly. Whats more they do exactly the same thing today as 20 years ago - nothing has changed except the increased effort to achieve it.
>Simple truth is that pre-Windows GUI (and now Web UI) these systems ran on simple Green Screen Dumb terminals (remember them?) they were more efficient, the staff were more efficient and the maintenance of these systems was simpler, more effective and less costly. Whats more they do exactly the same thing today as 20 years ago - nothing has changed except the increased effort to achieve it.
You may have a point in some cases but not in general. Worker productivity the past few decades has went up across the board like no other time in history and much of this is due to software interacting more efficiently with people. For example, the GUI would never have become so ubiquitous if it was not useful to the layman. In fact computers didn't really take off with the layman until the GUI came around. There are lessons to be learned from the past but the paradigm of hardware being much more valuable than development time as was the norm then doesn't translate well today.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
