back to article Australia finds $1 BEELLION to replace No-SQL DATABASE

That sound? The first drops of a cool billion Australian dollars (US$770m, £520m) that the nation's government has started pouring into a honeypot for the world's technology community. The sweet, sweet pot of dosh is there because Australia has decided it can't go on running Model 204, an eighties database, to run its welfare …

  1. DainB Bronze badge

    So without IBM how many of listed vendors actually have database product of that scale ? Answer is two. Funny enough they also manufacture their own hardware for their databases and providing consulting services.

    So how's HP, CSC and DiData deserved mention at all ?

    P.S. IBM might not be in favour anymore but Centerlink is very much IBM shop, so I'd say their chances are pretty high.

    1. Snow Wombat

      Given IBM's track record...

      It'd be insanely stupid to even bother with them.

      They have already been banned from QLD for gross incompetence and corruption.... and for Qld to do that, is saying something.

      1. david 12 Silver badge

        Re: Given IBM's track record...

        The [Commission of Inquiry] concluded ... the primary cause of the problems suffered by the Payroll System project was the State’s 'unjustified and grossly negligent conduct'.

        Sadly, the government wasn't banned from QLD for gross incompetence and corruption.

    2. Anonymous Coward
      Anonymous Coward

      They've got two choices:

      - Old-school single database instance with a really big server.

      - New-school clustered database with several small servers.

      The industry seems to be moving in the latter direction. IBM and Oracle are stuck in the old world.

      1. Nigel Campbell

        Actually, both Oracle and DB/2 can be deployed on server clusters. Although the architecture is still shared disk, a billion AUD is still enough to buy a few SAN controllers to partition the storage across - and any hardware up to a whole Sysplex to run the DBMS.

        Crikey - with that much they might even be able to afford Oracle's licensing.

        1. Anonymous Coward
          Anonymous Coward

          Crikey - with that much they might even be able to afford Oracle's licensing.

          Maybe, just… but won't leave much change.

        2. P. Lee

          >Crikey - with that much they might even be able to afford Oracle's licensing.

          Cue license change where everyone with Centrelink data is now an oracle "user."

        3. Anonymous Coward
          Anonymous Coward

          As can SQL Server be clustered - in both scale out and scale up. And for a lower TCO per transaction than Oracle or IBMs offerings.

      2. AndyDent

        "The industry seems to be moving in the latter direction"

        Weelll, according to Dave Thomas's presentation at the Yow West conference in Perth this week, that might change. He showed some very interesting figures about server affordability and performance from databases like KDB.

        End-user programming is going to come back and when I commented to him that it sounds an awful lot like the Pick system, he grinned and said Yup!

      3. Matt in Sydney

        MongoDB ?

        Indeed, and the growing popularity of similar databases - (MongoDB could carry M204 data quite happily) leads to more questions. Latest release of M204 has intriguingly included JSON features that seem to recognise this. It boils down to performance. As more transactions are end-user based (i.e. not employees) a performance hit is probably acceptable. The sad fact is that a lot of the complexity revolves around methodologies and processes. As with any workflow based system replacement, the workflows need to be addressed to really make a dent. Workflow performed for free does not matter so much ? In any case it is a lot of money to spend that could be deferred another 10 years and the saving could pay for the effort to maintain a current system many times over. It's not as if SAP or DB2/ORACLE are cheap.

    3. Mayhem

      While IBM *as a vendor* is out of the running, IBM *as a hardware platform* is a pretty strong contender because there will be a logical and reasonably well established migration path from the current hardware and database to wherever they want to go. The key part will be migrating all the customisations that have been put in place over decades, and *noone* will want to be writing them again from scratch.

      I expect that one of the major domestic IBM shops will take on the work with their own experienced engineers, along with various warm bodies contracted in from around the world for specific expertise.

      Upgrading a core system like this should be relatively straightforward, although I doubt that the people in charge have really thought out what they want done.

  2. Anonymous Coward
    Anonymous Coward

    So should be able to clawback a few million in savings from those "bludgers" every year then.

    Almost paid for.

    1. Anonymous Coward
      Anonymous Coward

      Gotta love those welfare "reforms".

  3. dshan

    It's called Model 204, not System 204.

    1. P. Lee

      Not this then?

      https://books.google.com.au/books?id=u7WbsmbttwYC&pg=PT192&lpg=PT192&dq=instant+recall+database&source=bl&ots=vQZyhDZU2L&sig=tdtoNqQF7CA2q2Fq3jzKpW-vQyA&hl=en&sa=X&ei=udApVeG2MsXRmAWX5YDQDA&ved=0CCcQ6AEwCQ

      1. Michael Wojcik Silver badge

        Not this then?

        I can't tell if that's a sincere question (and, y'know, as a badge holder you could have used an actual link rather than just pasting the URL as plain text). But no, the system in question uses Model 204, a DBMS from the late 60s or early 70s (depending on whether you date it from the research or commercial implementations).

        Among its claims to fame is, of course, its age - it's contemporaneous with IMS (IBM's hierarchical DBMS1) and a little older than INGRES (the first commercial RDBMS) and System R (the first SQL-using RDBMS). Beyond that, it offered extremely high performance on 1970s hardware, thanks to its use of bitmapped indices; starting in the '80s it also offered hybrid B-tree indexing. Presumably it's still fast on relatively low-powered hardware but typically hardware performance growth has outpaced data growth for Model 204-based applications.

        I've seen a few queries about Model 204 migrations from customers over the years. The main obstacle is the proprietary syntax, which means rewriting the data-access portions of the application; and these applications are rarely well-partitioned into data-access and business-logic layers. (Often they couldn't be, given the constraints of their original platforms.)

        I played around a bit with Database Programmer's Toolkit, a free Windows implementation of Model 204. Doesn't seem to be around any more.

        1IMS is more than just a DBMS, of course, but let's not muddy the waters.

  4. ForthIsNotDead

    Hmm...

    "The agency sends 180,000 letters and emails, and dispenses $290m, every day. System 204's doing that handily, but has reached the point at which meaningful changes are becoming tricky."

    You know what, if it's still ticking along, I'd be tempted to leave the bloody thing exactly where it is. I don't know about the system, if it runs on old antiquated hardware etc, but probably be thinking in terms of upgrading the hardware platform that the current system runs on (perhaps virtualisation, or emulation of the hardware) but try and leave the software itself alone. It's from the 80's, so is it an old Vax system or something?

    As usual, Wikipedia has the answer:

    [quote]

    Model 204 is a database management system for IBM and compatible mainframe computers, “born” 1965 October 13,[1]:66[2] and first deployed in 1972. It incorporates a programming language and an environment for application development. Implemented in assembly language[2] for IBM System/360 and its successors, M204 can deal with very large databases[citation needed] and transaction loads of 1000 TPS.

    [/quote]

    I would have thought IBM would have a migration path for old IBM System/360 stuff. Are they not the undisputed kings of backward compatibility? This would be an interesting project.

    I can just imagine the utter chaos that a new system (from the ground up) would bring, though. In fact, I don't even want to think about it!

    Edit: There is emulation available: (from the wikipedia page):

    Database Programmer's Toolkit is a freeware PC-based emulation.

    Haven't turned up any links though.

    1. Anonymous Coward
      Anonymous Coward

      Re: Hmm...

      "I'd be tempted to leave the bloody thing exactly where it is."

      Probably right. Sounds as though they want new software architecture, new data architecture, a new and fashionable operating environment, new hardware, and a revised set of application requirements.

      What could possibly go wrong?

      1. a_yank_lurker

        Re: Hmm...

        Better question, what will go right?

    2. lime

      Re: Hmm...

      Their comment was that it is currently working nicely, but it's getting difficult to make meaningful changes. What I assume this means is the code is complex, fragile and full of 30 years of hacks and workarounds.

      The brief will not be "provide us a new system that does exactly the same as our current system", but "provide us a new system that fits a new, different, simpler process."

      Whatever change they wanted to make to the welfare model could probably be done to the current system, but it would cost $xm and you'd still have the current system at the end of it.

      1. Benjol

        "New system that fits a new, different, simpler process"

        Hahahahhahah.

        That's what they all say, until you actually get down to brass tacks, then it's: "oh, we could do that before, and this too, and the opposite, and we absolutely HAVE to be able to carry on doing it that way".

        The cost is in the exceptions.

        But it's also what keeps us gainfully employed.

      2. Michael Wojcik Silver badge

        Re: Hmm...

        Their comment was that it is currently working nicely, but it's getting difficult to make meaningful changes. What I assume this means is the code is complex, fragile and full of 30 years of hacks and workarounds.

        Yes, that's generally the case with legacy applications this old. And you have to train staff to work on it - there isn't a huge pool of Model 204 expertise out there.

        That doesn't necessarily mean that reimplementation is worth the risk. But with an application like this one - which is probably exposed to the whims of the legislature, for example, since it deals with government funds - I'd be nervous about the cost of maintenance.

      3. FlatEarther

        Re: Hmm...

        'The brief will not be "provide us a new system that does exactly the same as our current system", but "provide us a new system that fits a new, different, simpler process.'

        Never heard of the second system effect, then?

  5. knarf

    System 204's deficiencies have been known since about 2004, but the government now says replacing it will “reboot” Australia's welfare systems, “ boost efficiencies and help advance many welfare reforms”.

    We call it austerity in the UK

    1. Anonymous Coward
      Anonymous Coward

      @knarf

      "We call it austerity in the UK"

      No, you call it austerity. Where sane people live, the UK bunglement's actions of spending £100 billion each and every year more than they raise in tax revenues is rightly regarded as insane profligacy that can only end in tears. If you want real austerity, go to Greece, and even there, with one in four of all workers unemployed, and one in two of all young people unemployed, they are still only managing to balance the budget (which is why they keep on needing new international loans to repay existing loans).

      So for every one of the UK's 30m taxpayers, they're going to have to pay the interest on £50k of debt each year (probably forever), and your "austerity" is adding £3k a year to each individual's personal allocation.

      1. Anonymous Coward
        Anonymous Coward

        Re: @knarf

        Of course most of these numbers - GDP, BoP, inflation rates - are just WAGs adjusted to taste. Personally I'm still peeved about the sterling devaluation in the 1960s that turns out to have been totally unnecessary since the BoP figures at the time were so badly wrong.

  6. Anonymous Coward
    Anonymous Coward

    Working a a similar system in europa we estimate the complete conversion to a modern system will cost about 50-100 million euro's if you do not extend the functinality but to 200 million or more if you want to extend functionality to things like client case based handling, workflow and electronic dossier system. The bill could run higher if big changes are made by the government in the benefits rules and laws during the transition.

    They should also expect operational cost of the system to triple or quadruple for at least the first ten years of operations.

    I would suggest using a basic functionality upgrade to a modern system that can be extended in future preferbly by more than one third party solution provider. Stay away from integrated solutions of Oracle, IBM or SAP as those look great on paper but eventually will run op your bills by hunderds of millions more.

    Build a .NET of java based solution on a generic modern database which you can much easier add different products to later even with other softwaresuppliers. Don't use dedicated ready made frameworks unless you know the vendor will almost certain supply it for 30 years more in the future

    1. Anonymous Coward
      Anonymous Coward

      erm... are you sure you want a .Net or Java solution

      ...you probably know better than me but I know of quite a few banks are replacing their .Net applications for Java... and as for Java itself, being owned by Oracle kind of puts it in their camp for the future... who know what Oracle will do! :)

      IBM has been a safe bet for them if its been running for the last 30 years no matter how fubar'd the politicians and project managers managed to get QLD, I suspect that changing the project scope as youre running through the delivery is not a good idea and that the consequences are built into the contract otherwise one side has employed a bunch of Jim Hendersons "Muppets" after their last movie.

      1. Anonymous Coward
        Anonymous Coward

        Strange

        Strange, because I know several doing the reverse. Java is a massive security hole to have to install on your clients, .Net is significantly faster in general and especially for calculations - and no one I know wants to build anything new these day on a platform supported by Oracle...

        1. Anonymous Coward
          Anonymous Coward

          Re: Strange

          Written in Java does not mean you require Java installed on the client device.

    2. Anonymous Coward
      Anonymous Coward

      "Build a .NET of java based solution on a generic modern database which you can much easier add different products to later even with other softwaresuppliers. Don't use dedicated ready made frameworks unless you know the vendor will almost certain supply it for 30 years more in the future."

      This is always a tricky decision to make, often re-creating the functionality in a off-the-shelf commercial framework will cost a few million if you're writing it from scratch. It really is important to decide if it's worth it.

  7. disgruntled yank

    Non-relational != flat-file

    Model 204 is not really a flat-file system. A friend used to work with it years ago.

    1. AndyDent

      Re: Non-relational != flat-file

      I figured it was either ISAM, network or hierarchical but few of the kids nowadays understand that anything powerful existed prior to "relational" databases.

      1. Michael Wojcik Silver badge

        Re: Non-relational != flat-file

        You can find some information about in a Google Books excerpt of O'Neil's presentation. There's a section on internals that starts on the bottom of page 49.

        It looks like it's implemented as an entity-keyword-value database, partitioned into a set of physical files. There's a direct mapping between record numbers and disk locations (the sort of "pointer" Codd was concerned with eliminating in RDBMSes). It uses at least three types of indices - bitmapped, hash, and B+-Tree - and is optimized for Boolean queries (which of course are particularly convenient to perform with bitmap indices).

        Certainly it lacks the consistency guarantees of a transactional RDBMS (for databases in normal form), but in terms of achieving fast query performance for large databases on limited hardware it looks very clever.

        It's typical of the sort of close-to-the-metal design that was common for high-volume data processing before the combination of Moore's Law Observation and the commodification of general-purpose computing made it feasible to layer any number of abstractions over our data access. Circa 1990 I took a DBMS course where we did back-of-the-envelope calculations for various physical file layouts on disk. These days that wouldn't even make sense, since so many layers exist between the DBMS and the physical storage.

  8. Anonymous Coward
    Anonymous Coward

    Won't somebody think if the Greybeards

    A lifetime of supporting obsolete crap without having to learn any new skills. How will they cope when they realise their ancient abilities no longer command premium prices and that they might have to adjust to a new tech.

    1. Tannin

      Re: Won't somebody think if the Greybeards

      ^ Spoken like a kid with wet ink on his tech certicate who thinks the answer to every technical problem is a newer iPhone.

      1. Anonymous Coward
        Anonymous Coward

        Re: Won't somebody think if the Greybeards

        On the contrary - 20 years in IT as a server engineer, dodging lazy old bastards with an entrenched, rose-tinted view of obsolete old shit, that could run for a fraction of the cost on newer virtualised hardware and a supportable o/s

        1. P. Lee

          Re: Won't somebody think if the Greybeards

          Those lazy greybeards aren't going to cost anything like $1bn.

          I don't have a problem with new tech, I do have an inbuilt suspicion of recent licensing agreements.

          For $1bn, I'd say, "I'll pay for the work, but I want it open-source."

  9. Michael Wojcik Silver badge

    Known since when, now?

    Model 204's deficiencies have been known since about 2004

    Since about 1965, you mean. I don't believe anyone ever claimed Model 204 would forever be all things to all people. Certainly since the early '70s people have been able to articulate the relative advantages and disadvantages of different DBMS architectures. When System R and INGRES showed that RDBMSes could offer feasible performance for many real-world applications, most people felt they had to admit that it was horses for courses.

    Perhaps circa 2004 was when people started questioning the economics of using Model 204 in this particular application.

  10. Ian Joyner Bronze badge

    Benefit all Australians?

    Morrison: "Coalition Government is getting on with the job of building a new system that will benefit all Australians." Hah, all this government want to do is benefit the rich and multinational corporations. They believe in the trickle-down effect. It is actually a bubble-up effect - money rises to the top, where it should be taxed and put back in at the bottom - that way it bubbles up again and everyone benefits.

  11. Ian Joyner Bronze badge

    They should consider Burroughs

    Oh, I mean now Unisys. ClearPath MCP are most solid machines ever. Proof - they are not built on C, but a proper HLL.

  12. Anonymous Coward
    Anonymous Coward

    Should Consider HP NonStop

    A good solution would be to implement a HP NonStop system, that support massive geographically distributed databases that can function as a single system. Their systems have been used for mission critical applications in medical, finance, defence, and retail where speed, and 24x7x365 availability is required.

    http://www8.hp.com/au/en/products/servers/integrity/nonstop/products.html#!&pd1=1

    http://h17007.www1.hp.com/au/en/enterprise/servers/integrity/nonstop/nonstop-sql.aspx#.VS2WGnAmKrU

  13. Anonymous Coward
    Anonymous Coward

    No doubt the whole lot could be done using sqllite

    In reality it could be handled by a single instance of (any)SQL... but to do it with some level of safety and scalability postgresql would be probably most sensible.

    Certainly there will be a lot of queries and third-party batch jobs, others running at the strange times and in a somewhat inefficient way. On top of that and least understood will be a total lack of required maintenance, let alone a single round of normalisation since the day it went live.

    That said, by doing a little maintenance they could stop whatever is causing the existing system to scale (fail). But no-one wants to do that in case it goes another 30 years without any love.

    No matter what licensing model they opt for (waste our money on), all it will actually need is an export to a shiny new schema, a well-implemented redundant database, some attempt at separating the private, confidential and public data into areas that are appropriately secured, and for someone that knows how to write code create some new access applications that implement secure, role-based access as well as fast data-entry and data-access under the processes that are presently carried out by each group of users.

    But what the vendors will do is shackle the back-end to layers of abstraction and duplication, lock it in to their hosting solution, start ratcheting up the costs year on year and begin building out some poorly documented APIs that may or may not provide performance depending on everything unknown. At the same time they'll build out some client side java to compound the complexity, maximise the attack surface AND implement some web-frontend(s) to make using it as painful or at least slower than it was to actually use the old system.

    Or something worse that invites a catastrophic failure in bureaucracy let alone technology.

  14. Anonymous Coward
    Anonymous Coward

    M204 is still alive...

    M204 is still quite alive (if not as widely used as once it was); http://www.rocketsoftware.com/products/rocket-m204

    (CCA was bought by Rocket Software in 2010; they also bought Sirius Software (the M204 extensions) in 2012)

    Given that, unless the developers applied considerable discipline, an M204 database may not trivially map to a relational database, it might be easier just to stick with what's worked. There are likely a few other government users, if one digs a bit, although they may not be the sort to advertise themselves. There's a free and open source clone that runs on Windows called DPT, but it's site has been down for a couple of years now, although a posting to the M204 User's Group Facebook page includes a link to a google drive stash of software and docs; I gather the author had hosting and/or budget issues. :-) It likely wouldn't be suitable anyway (except for training developers), since it lacks some of the security features of M204.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like