UK.gov still drowning in legacy tech because no one's boarding Blighty's £700m data centre Ark Only in IT is “legacy” a pejorative term, where it is used to condemn ageing systems and forgotten workarounds. In the UK government, as with banks, increasingly difficult-to-maintain mission-critical systems are a huge problem. Not least because of the dwindling number of folk who remember how the damn things work. One …
(This was first posted here: https://forums.theregister.co.uk/forum/containing/370244 roughly 8 years ago. Pardon the reference to the earlier thread in the third paragraph; I left it intact because it illustrates my point. The PDP11 code is still functioning as intended.)
Legacy stuff isn't the problem.
That's a myth.
There are running, functional systems, and there are broken systems. Legacy stuff that is still running, and functional is (by definition) not broken.
All of the examples listed in the 17 (at this moment) comments above are MANAGEMENT errors, and have nothing to do with the functionality of the systems being discussed.
The biggest management blunder is not having a plan for the future.
The second is not standardizing on functional systems (including upgrade path).
The third is not properly training staff in the use of those systems.
I have 30 year old machine code (running in a PDP11 emulator on Linux) that one client of mine is happy as hell to have ... It helps him keep track of a portion of his business that would otherwise require three or four pairs of technically trained eyes. (It's a specialty greenhouse, if you're curious ... the guy grows orchids, and keeps track of humidity, temperature, light levels, soil moisture, pH, salt, etc.). It originally ran on a pair of PDP11s (redundant systems ... hardware was flaky 30 years ago), now it's on a pair of dead-screen Pentium laptops that cost a total of $50. His electricity bill dropped, and the UPS powers the laptops for a little longer than the PDP11.
Is legacy stuff inherently "bad"? Nope. Bad management is, though.
(And then, about four days later: https://forums.theregister.co.uk/forum/containing/373033 )
Why are you trying to demonize "legacy"?
Serious question.
Old stuff isn't inherently bad. If it still performs the job it was intended to do, let it. If it doesn't, replace it. You DO have an upgrade path planned and budgeted for, right? No? Then I submit that it isn't an equipment problem, but rather it is a management problem.
I commented on my 30 year old greenhouse management software last time. This time, let's talk telephones. Palo Alto, that hot-bed of high-tech, still had analog switches in a couple of the local exchanges ... in 1998! They still worked for voice, and that was all the customers contracts paid for, so why change 'em? It wasn't until customers started getting upset that they couldn't connect to AOL at 28,800 "like my neighbor" that they switched to digital. The 326 exchange was the last to be changed over. If it wasn't for AOL, it would probably still be analog ... and some of THAT gear was installed in the late 1940s! (I was a NOC-monkey, lurking under Bryant Street and on Fabian(RIP) ... I'm probably going to have nightmares tonight over the memories ...)
Or consider TCP/IP. You use it every single day. Virtually everyone connecting to the Internet uses it for nearly every pointy-clicky intratubes delight. When was it first implemented? (See RFC 675 if you don't know the answer). It's a third of a century old, fer chrissake! Do you consider TCP/IP old, dusty, and in need of replacement? (Cue IPv6 fanbois ...).
Or how about the basic UNIX API?
Old stuff is NOT inherently bad, although there are bad installations. Bad installs are not the equipment's fault, rather they are the fault of the people controlling the purse strings.
"Why are you trying to demonize "legacy"?"
Because (as the article says) you're left with technology that few people understand. This is on top of the fact that you're missing out on recent/modern updates. It's called technical debt for a reason.
Blame management, blame the developers and architects, blame whoever. Blaming doesn't fix the problem. Fixing the problem fixes the problem.
C.
"Why are you trying to demonize "legacy"?
Because (as the article says) you're left with technology that few people understand
I respectfully submit that a self-contained industrial control system, written in machine code and running on an extensively and carefully documented PDP-11 hardware is fairly easier to understand and maintain. After all, all the necessary documentation and manuals, all the way down to the circuit board layouts are available if you need them, and the code is limited to a few thousands executable instructions - something you can actually follow by hand if the need arises.
A "modern" replacement will likely be measured in many megabytes, with hundreds of dependencies and millions lines of code altogether (and quite likely a few binary blobs here and there, of unknown provenance and function, which will only come to light when they break and you have to fix them).
So in the end you are replacing a system which few people understand with a system which nobody understands.
And this is the real cost of "legacy". Legacy systems become sealed black boxes. Legacy increases the size of the minimum element of replacement, upgrade, spec-change or tuning. And because that list includes replacement and upgrade there is a viscious cycle at work.
"I respectfully submit that a self-contained industrial control system, written in machine code and running on an extensively and carefully documented PDP-11 hardware is fairly easier to understand and maintain."
It has also stood the test of time. Unfixed bugs are either rare or have known workarounds. Most failure types are already known, with known recovery methods. Not a lot of "unknown unknowns" remaining there. Which is something industrial customers can appreciate. They tend to be surprisingly content with known bugs that are well documented, but quite wary of new stuff with lots of unkown bugs.
My point is in the SubjectTitle, diodesign.
Permanently fixing the problem doesn't have anything to do with technology.
Or, put another way, you can throw any tech at the problem you like, but the problem is going to remain.
Recent/modern updates is a strawman.
It may be technical debt, but I still don't use an iFad to control my CNC.
"Old stuff isn't inherently bad. If it still performs the job it was intended to do, let it."
But there is more to it than that...Legacy isn't demonised simply because 'shiny new expensive kit can do that 2 hour job in 1 hour and 59 minutes!!' it is demonised because it is usually flaky after a lot of use, it is usually insecure as it wasn't built with security in mind and assumed that nobody would be able to get through the front door of the network.
Your statement is akin to saying "DDT isn't inherently bad, if it still performs the job it was intended to do, let it" - i.e. It might still perform the job, but that doesn't mean it isn't dangerous and a very bad idea.
Your statement is akin to saying "DDT isn't inherently bad, if it still performs the job it was intended to do, let it" - i.e. It might still perform the job, but that doesn't mean it isn't dangerous and a very bad idea.
Yet we still use DDT under some circumstances.
it is usually flaky after a lot of use, it is usually insecure as it wasn't built with security in mind and assumed that nobody would be able to get through the front door of the network.
Whilst I will admit that older systems may be less secure than more recent ones that does not usually prevent them from being isolated behind interfaces to protect them from the nasty outside world.
As for getting flaky after a lot of use? The software certainly doesn't get flaky. Hardware may fail and storage may exceed its intended capabilities.
'flaky after a lot of use' is a bit confusing (confused?). Any government system, by definition, will have changing requirements. That means the software has to change. Chances are that this introduces more dependencies within the code and lots of bits that are probably unused but left alone 'just in case'.
You can only extend a house so far. Eventually, you have to knock it down and start again with new foundations. Same goes for software.
"You can only extend a house so far. Eventually, you have to knock it down and start again with new foundations. Same goes for software."
I dunno. Look at Westminster (The church[0], not Parliament ...). Or York Minster.
[0] Yes, it's a church, not an abbey; nor is it a cathedral.
I think you'll find that under the current York Minster are the remains of 2 previous churches plus a roman building.
You'd also struggle to extend it much further as the foundations are barely good enough and there are other nearby buildings preventing it being enlarged.
"I think you'll find that under the current York Minster are the remains of 2 previous churches plus a roman building."
I think you make Jake's point. The only reason the earlier buildings are known is because the central crossing area was excavated to replace the foundations whilst leaving the building intact.
"Any government system, by definition, will have changing requirements. That means the software has to change."
That's the great thing about supplying the public sector. If the politicians could just manage to sit on their hands and quit meddling with things for 5 minutes, I'd be out of business.
"Eventually, you have to knock it down and start again with new foundations."
Comparing houses with software would require not only to knock down the existing house but to knock down the replacement structure(s) ten or twenty more times before a self-standing structure is finally built.
but to knock down the replacement structure(s) ten or twenty more times before a self-standing structure is finally built.
Then knock it down and replace it everytime any of the locks, windows, curtains etc. need to be replaced or it needs a coat of paint...
> Hardware may fail and storage may exceed its intended capabilities.
Having two ancient (90s) PCs fail within a relatively short space of time, the issue of system (as opposed to hardware) longevity is at the fore - as long as the thing looks the same from out there, nobody cares if it's soundproofed with bits of old carpet, rubble-proofed with stapled together bits of plasticised cardboard...
Saved by both having add-on cards to handle HDDs larger than 8GB (due to previous temporary death, and card+disk cheaper than a rare 'legacy' disk) and the card+disk could be bunged into a newer (early 00s) box (P4, socket-423) and thanks to the mixed blessing of 'generic x86', run with no changes except a different NIC ID. Newer hardware means I don't technically require the custom boot CD I had to make (old hardware could not boot via add-on card) but right now it works so it stays as-is... this is where 'legacy' comes from.
P.S. an upvote for AC above "So in the end you are replacing a system which few people understand with a system which nobody understands." to which I would add 'and has a licence to fix'...
I respectfully submit that a self-contained industrial control system, written in machine code and running on an extensively and carefully documented PDP-11 hardware is fairly easier to understand and maintain
Surely it is. But that does not change the fact that software which can only run on a difficult to replace piece of hardware is itself at the risk of becoming non-functional at any moment, due to said hardware suddenly becoming non-functional itself. That's where the management role is to look for replacement, to remove that particular risk. Of course one needs to have deep understanding of the subject not to allow to get dragged down a multi-million-LOC project, and that's another area where managlement is known to have weak points ...
I'll bite....
I couldn't agree more with "it ain't broke, don't fix it". However when it does break you try getting parts of a 25 year old piece of DEC kit other than some dodgy seller on eBay who just happen to keep the kit in a damp garage for the last 5 years. You get it cheap but it might only buy another 2 years max? What about installing that old kit? You try finding anyone under 35 years old who's worked on any other kit than bog standard PC? I'm nearly 50 and the first 5 years of my IT career back in 1987 I worked ICL System 25, DEC alphas and MVS, then RS6000, a bit System70...then for the last 20 years it's been all PC ( Windows and Linux ) and Sun SPARC kit, nothing else. I can barely remember how to init an ICL System 25 box let alone reinit disk for storage pool allocation on a SYS25!
IT has to move on, we don't like it but once I retire I couldn't give a monkey's! No amount of money will get me back to coding COBOL or rebuiling AIX systems, I simply don't care and I know dozens of others who want to quit IT now, do their hobbies and have an easier life. They don't want to be fixing legacy systems. So when those systems go down, no one will be there to fix them,
The definition of Legacy is anything one set of people want to replace with something else, regardless of its real value capability and supportability. If you attach the word Legacy to anything, it automatically implies it is outdated, expensive and unsupportable, regardless of the truth. A well known company refers to anything that isn't their software as Legacy, can you guess who.
There are several separate issues here.
One is that the system is currently earning money. Where's the pay for all those dev[op]s working on shiny new systems coming from? Probably from that despised legacy system.
Then there's the age of the system and its state. There are a couple of assumptions being made. One is that it's old and the second is that, simply because it's old it's ill-maintained. Neither is necessarily true.
For one thing there was an article on here a little while ago ( https://www.theregister.co.uk/2016/06/16/online_graze_in_reverse ) about one relatively new business deciding its whizzy web-based system no longer suits and has to be replaced. This isn't stuff conceived in COBOL on a 360 but it's still legacy.
It's also by no means certain that something that was first put together years ago hasn't been maintained properly in the interim. If it's running the main line of business there's every reason to make sure its fit for purpose. If it isn't then its maintainers haven't been doing their jobs right.
The problem with attitudes expressed here, and maybe with legacy systems if they're not well maintained is that new development is seen as important, challenging, rewarding and whatnot, able to adopt the latest buzz word methodologies of agile and devops. Maintenance is just maintenance and, if done right, involves serious thinking about how to graft new stuff in seamlessly and keep documentation up-to-date. Actually it can be more challenging and rewarding if done right but it's apt to be seen as somewhere where those least able to do it well get pushed out of the way. Which is exactly the wrong way to treat what's paying everyone's wages.
It's easy to blame civil servants for cocking up IT contracts, projects and provisioning, but they are generally better qualified and more knowledgeable than their political masters (I know: that's not hard).
The core problem is that politicians utter these grand verbal farts like "Do this big job all at once and save tons of cash" and "Don't bother me with the details, after all, how difficult can it be?" knowing that they'll most likely be moved to another ministry before they've even learned anything worthwhile about the current one. And because they are all style, spin and mouth, without substance, they are *never* the kind of sensible people who understand details or even why details matter.
So they leave one disaster after another behind them - and because IT is visible and measurable and hard to lie about (unlike, say, disability benefits testing) the catastrophes become embarrassingly public.
In government as in corporations, the eternal rule applies: bad management, like shit, rolls downhill.
I did some analysis for the technical requirements for a project with the Foreign and Commonwealth Office. The senior manager was a diplomat. A very nice chap but without even half a scooby about IT or requirements. I was stunned to find out that he was only in place for a couple of years (planned), this was a stepping stone before he moved back into diplomatic duties. So the result of this policy is that the person with ultimate responsibility for determining the spec and getting the project underway was not the person running the mid point (poor sod took on a duff project) and the individual seeing full deployment was different again (although at least he had the comfort of being able to blame the last two duffer - who had moved on of course to better things) was different again!
Of course ur diplomat was frequently seen being wined and dined by the company and no doubt will move on as an advisor or board member in future years. Moral hazard writ large (as are the armed forces and many others in government/state run services)
"Well I have observed, that just like scum, bad management frequently rises towards the top."
It's called the Peter Principle.
Unbelievably, most of them actually started off good at their jobs.
Then they kept getting promoted until they reached a level they're not fit to handle.
Then rather than admit or even acknowledge they made a mistake, their boss just keeps shuffling them around from position to position to hide the damage.
Why would their boss do that? How do you think he got where he is?
a) The fact there's no super IT dept that can go round and boss the others around so there's no coherent IT strategy
or
b) the fact that the other departments in all probability looked at this Ark thing which wasn't cheap to set up and whose board was just an old boy's network and in all probability found it wanting?
"how the hell are you supposed to replace it with a modern version?"
Easy. When the legacy h/w falls over and dies, and no one will replace or repair it, then you have to bite the bullet and put in a new system. Factory control systems are a different animal, they can routinely be left disconnected from the Internet and thus remain relatively safe in that respect. And they are generally so expensive that there will always be someone around to offer you expensive hand-holding. For business systems and other general purpose compute hardware the problem is; "is it worth spending money and having possible downtime to update a system we know is old, but is being used heavily and can't be interrupted?" Because that's what it boils down to; money. It takes money to purchase the new system and to hire people with enough skills to properly spec the legacy system to tease out all the nuances to how it works and how to replicate it. It can be done, and you don't need the original designers or support people to figure it out. Any black box can be opened up, inspected, and replaced. Once replaced, then the paranoia about "how do we get support when we fire the people who care for it now?" becomes the next big issue. And that usually ends up being the other expensive part of the solution. That's why I prefer to offer open source solutions and properly document the build, so future admins and support people can figure out and repair/replace what we did "way back when." AND my solution did not cost a princely sum for support on what should be a thing that never needs service, once the spec is complete, for the most part. I'm making a wide generalization, but the core of the gist rings true, if you've worked around "legacy systems" at all.
This reminds me of what I call "frozen systems" where I decide that "okay, there is a limit to the ability of this hardware to properly run any new builds" because of resource concerns (like when Apple releases a new OS. I can't install that on my ancient Mac, there is simply not enough power left in the old kit). Also when I decide that a system has a cost to the upgrade that it is not worth the cost/effort to make it new, as long as it keeps providing services. And if some software has a final version I will use that to kind of "top off" the package/OS so it keep fresher longer. But to upgrade without a need is something I'll never do.
I can't see any reason why HMRC (for example) would want to go to the expense of physically relocating whole data centre halls of kit - out of it's domain of control.
Also once you've virtualized say a WS2K3 instance, I don't see any benefit in running it as a VM in ARK's data centre, in fact I see lots of negatives.
Finally, I suspect any mainframe vendor would recommend buying and installing a new mainframe at ARK, rather than risk shutting down and moving a fully operational system, which would only add further to the costs...
One of the goals was to split the hosting contracts from the technical management contracts. So they could (for example) move management of the kit from supplier A to supplier B without having to physically move it between supplier Data Centres. It's a flawed goal, but that was the goal.
AC Obvs...
are huge, unwieldy things that take many years to understand let alone transform into code. And I'm not talking software here, I'm talking systems as in ways of working. Every year some tweak is made to departmental budgets, tax codes, legislation etc etc. And every six months some tweak is made to operating systems, hardware, interfaces etc. It's like trying to hit a moving target from a moving platform with a wobbly crossbow.
As long as you understand that, you could probably project manage your way out of it. But it would take a team with minds like corkscrews to work it all out and devise a system as flexible, resilient, secure and future-proof as required.
Our devops guys are moving legacy systems (from a variety of external teams) to a virtualised, centralised data centre at the moment, and I've never heard such a volume of terrible language (and our devops/sysadmin team of bitter, hateful linux guys are not renowned for being polite at the best of times). 'Which c**t thought this was a good idea?', 'Hey guess how many processes this sh*tty f**king script generates? It only doesn't take the machine down because there is another f**king bug on this mailserver!', 'Why are the comments in this script in Hungarian, how do I say c**t in Hungarian??'
No one's using Ark because it's ungodly expensive by modern standards. It was specced up for critical national infrastructure apps, the crown jewels kind of stuff that only really HMRC and a small group of others run (hence 'bloody great army base'). Unfortunately the business case was built on huge swathes of the public sector (including local authorities, police, transport etc.) taking advantage of Ark's services.
Ark is only cost-competitive for their most critical of apps, so short of investing hundreds of millions of quid decoupling their systems from one another they're simply not going to jump ship. They'd have to move their entire estate into Ark, paying more than they otherwise need to for gucci service for non-gucci apps (itself a non-trivial ask anyway).
These departments and bodies are all going through the process of finally, after literal decades, breaking free of their monolithic infra contracts (often from Fujitsu via way of ICL). They're not in a rush to invest nine-figure sums to jump into bed with another monolithic supplier.
The other side of the problem is that the service offerings are naff. Ark looks and feels like an AWS-lite, with none of the add-on services. You've still usually got to partner with another supplier to actually manage your Ark kit (e.g. Skyscape, Vysiion, UKCloud) and what you end up with is bad Openstack layered on bad VMWare on bad kit. And heaven help you if you need a service that isn't in the offer book to start with.
"If it an't broke don't fix it."
Absolutely true. Unfortunately, at some point it will break. And then who will fix it if no-one understands it? Much better to plan for a phased upgrade/replacement while it is still working, rather than panicking when it's realised the last person who did understand it has shuffled off this mortal coil.
"Unfortunately, at some point it will break."
Why? Do the bearings wear out because of all the iterations of loops? Do they start out with a finite stack of branches to take at branch points and finally run out? Or does some unskilled maintainer, who shouldn't have been let near break it by trying to fix what wasn't broken?
*> Ministers only get to make big spends if it claims new shiny kit/services, not on maintenance.
*> Maintenance/replacement hindered by sourcing deals for lifetimes longer than supporting IT with no provision for maintenance / replacement
*> System complexity increased by multiple integrations & data exchanges
*> Integrations & data exchanges are not via any kind (never mind standard) service bus or integration layer
*> No system is ever changeable by the commissioning department once integrated (everyone integrated needs a hand in it) making it impossible to fund (i.e. departments cannot subsidise each other's IT (or at least can't be seen to)
*> BAU funding is so minimal there isn't usually and maintenance roadmap developed as no-one to do it.
*> failed projects for new/shiny or system replacement knock on to the retention of the systems they are intended (to partially) replace
*> Service availability/DR/reliability is not part of minimum viable product given the lowest price wins contracting model.
We all loose (except the big suppliers of course)...
"The Register whipped out the Freedom of Information Act to ask for a list of public sector bodies that have signed up to the arrangement. However, we were told the Cabinet Office could not disclose the customer list for this commercial arrangement because “authorities could be targeted by individuals or groups willing to use malicious or other hostile ways to gain unauthorised access to information (sensitive or otherwise) stored at collocation sites.”
Sounds like they think their systems may be vulnerable.
"One contact said part of the problem with Ark for government use was the fact that some of the departments’ legacy kit won’t fit in its racks, while in other cases the hardware is partly or fully owned by a system integrator – making it difficult to shunt their kit somewhere else."
You're gonna need a bigger rack.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
