"It is a sad tale of missed opportunity. A senior civil servant with no experience of running a complex IT programme; an overbearing Government Digital Service with misplaced philosophical goals..."
..And bears, they shit in the woods.
The head of the agile company instrumental in the UK Ministry of Justice's disastrous Common Platform Programme has stepped down. His departure follows a Register exclusive exposing the project's failings. Jeremy Renwick, chief exec of Agilesphere, held the role of Programme Manager/Agile Coach from February 2015 until April …
Using agile to deliver a government deliverable cannot work.
Procurement, finance, contract selection and deselection, etc are as waterfall as waterfall gets. Add to that the inevitable pre-election "what we have delivered" fixed dates and you get an environment where Agile has no logical function.
You can use agile as much as you like internally, but the actual shipment of stuff to gov (of any shape and form) and gov's side of planning the programme is not agile territory. It cannot be.
So the person to fall on its sword should not be the Agile consultancy, that should be the mandarin which decided to go all-buzzword compliant and run it as agile.
Using agile to deliver to government can work, and does. I've done several Agile projects for government and they've all been successful.
Even with a legislative deadline to meet, Agile is beneficial. It might look like a legislative deadline means fixed schedule and fixed scope, but in reality it doesn't mean fixed scope. The legislation will tell you what you have to do, but not how. With Agile the customer has constant visibility of progress, and it should be clear to them whether you're on course to meet their goals, or whether scope needs to be reined in and they need to settle for a simpler solution.
With waterfall, the customer's taking your word for it that everything is on course. They hope you're being honest and realistic with them. You hope they'll be pragmatic with you when you tell them requirements need to be cut (good luck with that if you committed to a fixed cost, fixed scope waterfall contract).
This failure probably isn't an agile vs waterfall thing. If they get 30 months into an "agile" project and all they've delivered is a glorified appointment booking system, clearly they haven't been doing agile very well. I expect if the same team had tried to do it as a waterfall project they'd have failed at least as badly.
"....With waterfall, the customer's taking your word for it that everything is on course...." Male bovine genetalia. With proper waterfall you get a comprehensive reporting system that keeps the customer well-informed. What happens is this - the customer produces a wish-list of airy-fairy nonsense as a requirements doc; the experienced waterfallists say "no, doesn't pass the feasibility study, go away and map out your requirements properly if you want the project to succeed", but the hipsters say "hey, no probs, we'll just go agile (and make it up as we go along)"; management ignore the waterfallists and accept the pipe dream that agile can solve everything; project fails; management finally bite the bullet, tank the hipsters, and ask the experienced waterfallists to try and rescue something from the wreckage. Seen it in government projects plenty of times.
"With proper waterfall you get a comprehensive reporting system that keeps the customer well-informed."
Who produces the reports?
Unless they're receiving actual functionality that they can use and test themselves, they're taking on trust that your progress reports are true.
"Who produces the reports?...." That depends on how savvy the customer is. If they are smart they will have one of their staff included in the team to make sure what is reported actually matches real progress.
"....Unless they're receiving actual functionality that they can use and test themselves, they're taking on trust that your progress reports are true." LOL, it's called a court case. As it was once put to me by an experienced PM, lying to the customer is bad, but giving them written evidence of your lie that could be used in court, well that's just downright stupid. The Government has lots of lawyers, time and money to pursue those that are stupid. Any consultant worth his money will make sure any report doesn't put a rope round his own neck by falsifying results. Consultants not worth their money....?
So true, speaking as someone who previously did system engineering for a public facing Gov organisation. Turns out most of the techies were doing Agile-like within their teams even before the high and mighty picked up on those trendy words. These are the same people who would not allow us to take up a discounted offer from a software supplier and make us go through procurement and end up paying more than even the standard price (as they went with a third party to supply us the same licence) and cause a delay of 6 months on a project.
Well, they have to, thanks to Lord Maude & pals, regardless of whether it makes sense or whether anyone knows what they are doing, else they don't get past the Guard Dogs & Sentries: https://www.gov.uk/service-manual/agile-delivery/agile-government-services-introduction
If you're a half-decent developer you're already more than enough 'agile' without the need to graft a religion on to what you do; if you're not half-decent, then no amount of consultants and twat-speak will make you better. Instead, try using your brain, or try a different career.
Sounds like you dont know what Agile actually is. It goes beyond the work of one single developer.
The whole point of Agile is to deliver small amounts of work on a very regular basis, it doesnt have anything to do with the individual talents of any one person - much in the same way as every project methodology doesnt rely on this.
How can that possibly be true? Good software has everything to do with individual talents. A lot of software is in the state it's in today because people like you think everything can be commoditised if you throw enough consultants at it.
"The whole point of Agile is to deliver small amounts of work on a very regular basis"
My experience is that there's a fairly substantial minimum of work that must be delivered to be of any use at all. If your first delivery is, say, a simple order entry screen and a database behind it there's no use anyone entering any data because there's no way of using it because the warehouse needs to be able to get its instructions from the system.
And if the next delivery is to print a despatch note it's still no use. Maybe the third delivery will be a picking list print so it's usable.
Then when the invoicing is added someone realises that the data collected is inadequate. Then go back to the beginning, revise the database and order entry.
A few iterations of this nonsense and you get confronted with an angry DBA who wants to know why you have to keep buggering about with database reorgs and can't you get a simple thing like designing a database right the first time.
That first delivery *is* useful because although it can't be used it can be tested and accepted. A sprint - especially at the beginning - doesn't have to be shippable, it just has to be testable. It may take multiple sprints to get to the minimum viable product but each sprint along the way should deliver incremental benefits - an interface, a database table, a screen layout with accepted UX.
And as for the DBA, just buy him a pint and he'll shut up!
@Mike Timbers
I work in Operations/system management.
Our experience with Agile projects is similar to the DBA reference.
We've had to rebuild 10 servers across multiple environments 2 times (in addition to the original build) because the Agile team has gone off half-cocked, had servers built for them for how they like to work, then come to us and said "oh, and when we go into production, you have to manage/support these systems". They used their own deployment system, filesystem layouts and so on.
So our first 'rebuild' (2nd build total) was to set the systems up in the standard we had that just slotted into our existing deployment methodologies and support/monitoring frameworks. And of course, since they were iterating through these sprints, detailed requirements were always changing, from environment-to-environment (dev, testing, integration, production), and since they had drop deadlines we had to, effectively, hand tune each server, so no 2 servers were exactly alike.
So after we got a couple drops into production, so we'd finally been able to nail down exactly their final (well, to date!) configuration requirements, we are rebuilding again with a consistent build/configuration across their servers, so that all are built the same way. So that's the 2nd rebuild, for 3 builds total.
The issues with your "just buy him a beer" are:
1) We don't have the staff resources to do that much repeat work for 1 project when there are dozens happening, we just don't have the manpower -irrespective of the ACTUAL budget- to perform that much work dedicated to 1 project. Doesn't matter how much money they give us, we have an allocated staffing level and we are not allowed to exceed (hire more staff) that irrespective of how much money a project team will throw at us (a projects budget is transient - capital, whereas hiring staff is a permanent employee, an ongoing operating expense, so the project money might fund 2 more staff for 1 year, but who's going to cover those staffs wages when the project is completed? Therefore we can't hire based on project funding).
2) They expect us to do work for them on-the-fly, to immediately respond to their emails/calls/IMs since they are 'agile' and need work done for them to complete this weeks sprint. This basically assumes we've got nothing else to do other than sit their waiting for their calls to do work. Whereas we may be
busy doing work for a project that submitted it's work request 3 weeks ago and have no time to work on their shit - until they cry and whine that we are the reason they can't make their sprint/release objectives and manage to find someone senior enough to either tell us to do their work as higher priority or increasingly often as my manager's manager is starting to do - tell em to fuck off and submit a proper work request and get in the queue behind everyone else
The problem is, Agile seems to require a massive increase in staffing level to support their iterative - i.e. read redoing the same work multiple times (a least in the infrastructure/middleware/backend sides) - and quick turn-around - i.e. expect you to be able to drop anything else you are doing and do work for them - approach because they haven't provided an end-state DB/hardware/support/operations/monitoring/capacity requirements that we can plan around and just build once and mostly forget about.
So buying the DBA (insert here other backend-type staff, infrastructure, server support, etc ) a beer is going to do fuck all for all the extra work and stress the agile teams are throwing around.
@edlakka, sorry but it's not agile that's at fault in your situation.
your agile teams didn't work with you at the beginning to agree the server build.
or
your agile teams could have developed their software, their table layouts etc. then worked with your teams to implement the service once the solution was functionally complete
or
your agile teams could have built their software to not have dependencies on your builds
or
your agile teams could have deployed onto cloud platforms and supported them themselves
Agile isn't at fault here because agile doesn't specify any method beyond iteration and constant checking with the stakeholder as the iterations progress.
You're describing a failure of process agreement, maturity and delivery.
This government project has failed precisely because of ITIL jockeys like you in very senior posts. People who do not realise what cloud and agility REALLY means. YOU should 1) be able to provision a box in minutes. Clearly you can't. 3 rebuilds? Are you fucking joking? It should have been closer every single mother fucking deployment. 2) Your box configuration should PARTICIPATE in the entire pipeline flow. I'm certain this is not the case either with your static shitty infrastructure.
I spent my career wondering how to turn science and engineering graduates (whom one would expect to think logically) into competent programmers. That is, produce programs which handle correct cases correctly, and error cases with error handling that does not crash. Programs which were tested to see whether they fail, not whether they barely scrape through a cursory test or merely compile.
Those skills cannot be imparted by management.
I never did find the answer.
Because it allows cock ups like the scheme this article is about and the reason why the emperor has been called out for having no clothes.
I have seen many areas where secure thinking is overruled in the name of delivery leaving the support function to pick up the pieces. With decent design and control this situation is avoidable, but in the interests of cost reduction this seems to go by the wayside. Agile allows people to forget tasks while they move (or are moved) onto other tasks and more glory.
It's not a personal thing as waterfall can either be like a trickle from the meltwater that is management or can result in a whirlpool that everyone gets sucked into.
I'm a taxpayer, voter and citizen and I expect my data always to be secure.
By allowing this project to fail are you saying the government doesn't take security seriously?
Security should be the first and foremost consideration of any project whether government or not. I've found this approach ensures projects are more likely to achieve their objectives if this approach is taken.
Agile does work and saying it doesn't is ignoring vast numbers of successful agile projects. It's like saying that waterfall doesn't work because of all the failed waterfall projects (and let's be honest, there are some fantastic examples of government waterfall project disasters).
This scheme seems like it was doomed no matter how it was delivered because the whole point of agile is that it delivers iteratively and continuously. If there isn't a step forward for more than a couple of sprints, a good agile project would be asking why and doing something about it. It sounds like this project simply didn't follow any governance. That's not a methodology failure; it's a governance failure.
> Agile doesn't work Because it allows cock ups like the scheme this article is about and the reason why the emperor has been called out for having no clothes.
um no. Thats not agile allowing cockups like this, its management/stakeholders.
The projects been running for 3 years. With agile (assuming 2 week sprints) that means management have had 78 demos/releases showing a complete lack of work being completed, and they ignored them all.
Waterfall would wait for 3 years and then give one release showing a lack of goodness.
No project methodology works.
33 years of experience suggests to me that well motivated teams of talented developers working with good management will produce good quality results in a timely fashion while demotivated[1] teams of mediocre[1] developers working with mediocre management will produce poor quality results late.
In both cases it will be in spite of the methodology adopted rather than because of it...
[1] Trust me, in the absence of competent management if they aren't at the beginning of the project they will be by 6 months in...
Sounds as if they weren't actually doing Agile at all but merely spouting the management bullshit that can surround it.
The projects I have been a part of that did use Agile were all about delivering working software first and foremost. From what i can remember that is pretty much top priority in the original manifesto.
Yes, no design, no documentation, minimal testing. Just working software. Yeah, right.
The problem is that although all developers think they can do that, very few are actually capable of it. The inevitable result is a large quantity of buggy poo, with a few decent bits here and there that the competent developers worked on.
Yeah, so an agile team comprised of shit developers will deliver crap code just like a waterfall team composed of shit developers will. The question is whether a competent team work better under agile or waterfall. Given the correct environment (ie. proper backlog management) an agile team of competent developers will out perform the same developers doing waterfall. This is because agile done properly allows developers the freedom to manage their own effort.
So many posters on here effectively saying "I can't be trusted to organise myself"....
Yes, no design, no documentation, minimal testing. Just working software. Yeah, right.
That isn't Agile!
Agile simply asks you to prioritise working software over complete documentation. Nowhere does it say that the documentation should be absent, nor does it place anything other than working software above documentation.
It should be patently obvious to all that failing to go through a conventional design process never produces anything of value; it becomes development-by-evolution, and almost all evolutionary experiments become extinct.
An old colleague of mine had a wonderful phrase: "A week's worth of keyboard-bashing can sometimes preclude the need for an hour's thought". That pretty much describes the approach of so many people who claim to be "Agile" (and clearly aren't).
Vic.
"Sounds as if they weren't actually doing Agile at all but merely spouting the management bullshit that can surround it."
P much.
I've done formal Agile processes in government (usually some form of scrum) and it goes one of two ways.
1) You're working directly for a small, operational team who just want shit done and you're able to deliver on time every couple of weeks. Nothing is perfect but everyone is happy because real things are being delivered, until central IT take over and it all turns to shit as they graft your delivery process onto their corporate standard. From then on you spend more time exporting JIRA reports to their standard project reporting spreadsheet template than you do producing useful stuff.
2) You're working on a "strategic initiative" or "digital transformation" with dozens of consultants where no one has any fucking idea what is to be delivered or what the overall goal is and there are dozens of disparate projects organised into programmes competing for funds. Senior leadership spend their days being bamboozled by the supplier and firefighting one thing at a time. GDS swan into the project once a quarter to tell you you're all shit and should use Ruby for everything. Nothing serious gets delivered but everyone gets paid because it's all T&M because government has *no idea* how to structure a contract to support iterative delivery.
It's not fun, but neither is it unique to agile or unique to government.
"1) You're working directly for a small, operational team who just want shit done and you're able to deliver on time every couple of weeks. Nothing is perfect but everyone is happy because real things are being delivered, until central IT take over and it all turns to shit as they graft your delivery process onto their corporate standard. From then on you spend more time exporting JIRA reports to their standard project reporting spreadsheet template than you do producing useful stuff."
Not to be cynical but it sounds like everything is fine while the development team asses its own progress and outputs but as soon as the software is used and assessed by someone outside the bubble of self congratulation reality bites but instead of accepting what was delivered did not meet the customers needs the customer gets blamed for not drinking the coolade.
When the customer is sat a handful of feet away from you using the stuff you're building and shipping once every week or two it's really rather hard to make the argument that the dev team is "assessing its own progress" or that the product "did not meet the customers needs". I would further propose, based on experience, that the delivery teams and end users are far better placed to report progress than a professional project manager whose sole purpose in life is to change the colour of boxes on his ever-sliding Gantt chart.
This is actually at the root what most Wagile frameworks get wrong - particularly in government. Someone takes on the title of "Product Owner" but they're not empowered to make decisions or knowledgeable enough to represent their peer stakeholders. What you'll usually end up with in the civil service is some mid-level fast-streamer from the IT delivery organisation acting as a classic business analyst. You end up without the iron-clad requirements and specifications of proper waterfall and without the empowered decision maker required by agile processes.
What you need is a decision maker with enough clout to bring round a consensus. Doesn't matter if the consensus is wrong - you fix it in the next iteration. It does matter if you spend 2 years and £200m dithering over requirements to the point nothing ever gets built, fooling yourself that it's ok because "it's agile", during which time the carousel of fast streamers and ever-changing senior senior civil servants has been refreshed four times over.
Guess which one happened here.
Yeah agree - powerless (and probably 95% absent) product owner makes a huge difference.
To all the waterfallists: you'd be surprised how similar Agile and Waterfall are in achieving results, if both are done properly. The main difference is Agile will deliver what you need now, whereas Waterfall will deliver what you thought you needed two years ago, and now have to lawyer up to argue about what features have or haven't been delivered.
>The main difference is Agile will deliver what you need now, whereas Waterfall will deliver what you thought you needed two years ago
Not heard of change control?
From my experience, the primary difference between Agile and waterfall is in the contracted deliverables and the ownership of risk. With waterfall, I the supplier commit to delivering a complete system that satisfies the requirements within a defined timeframe. With Agile, I the supplier commit to delivering some functionality through a number of development iterations within a time period.
"Not heard of change control?"
Change control: the act of trying to wrestle reality into being what you decided it should be two years ago, instead of evolving and adapting to reality as it is now.
"From my experience, the primary difference between Agile and waterfall is in the contracted deliverables and the ownership of risk. With waterfall, I the supplier commit to delivering a complete system that satisfies the requirements within a defined timeframe. With Agile, I the supplier commit to delivering some functionality through a number of development iterations within a time period."
You are comparing apples and oranges. The waterfall situation you describe is about delivering a defined outcome. The agile situation you describe is just T&M delivery of services, not delivery of a defined outcome. Agile can be used for both, but agile has a far better chance of delivering to expectations because of far better risk management abilities - unless, of course, management only pay lip service to agile without actually being agile.
"The projects I have been a part of that did use Agile were all about delivering working software first and foremost."
At which the users gaze in desperation because there's no documentation on how to use it.
And the developers successor s throw it away and write something that they hope does the same thing because there's no documentation for them either.