Culture, schmulture. DevOps, agile need to be software-first again

I enjoyed reading this, but...

As per the title, I enjoyed reading this but like most evangelical treatises about DevOps, Agile and such like, they all seem to refer to some form of utopian dream or of how these "should" work in the workplace rather than providing hard and fast case studies of actual issues and how they were overcome; and when I say that, I mean in relation to the transformative elements of totally unique and individual fully global integrated ERP solutions for example (in which I include legacy, mainframe, web and all the other components and teams that make up that solution) or to a two man business in a shed somewhere, rather than just seeming to expect everyone to be and work like Tw@ter or Facebook.

I know it sounds dull, as everyone mostly wants to make themselves sound smart by talking in acronym, or focussing on all of the glittering sparkly rocketship technology, but I absolutely support the initial view as to why "culture" is the most oft spoken about piece. This is for a very good reason. Get this wrong and you may as well go and burn all of the cash in a field.

Where I am currently we're in the early phases of implementing these newer methods such as Agile and DevOps - and I'm more than happy to be supporting this. But to do it the supposed "right" way (and I personally do not believe there is a one size fits all "right" way) despite what all of the pointless books and so called "consultants" say, would mean rebuilding the company organisation, people, hierarchies, systems, processes (Business + IT) and almost everything from scratch. Which just isn't going to happen.

Each company is unique in the way it works. Whilst the overarching theories about how to "rapidise" stuff is not, how they are implemented most definitely is, and how that "culture" changes (or cannot change) to absorb them is probably the most important aspect of it.

New tropes...

Agile - do away with subject matter expertise and have 'users' (salesmen, not users) talk to college kids about business needs that neither of them understand, regulated processes that neither of them have heard of, and industry standards that neither of them care about.

Sprint - smash out any old piece of software onto a shared test system and call the story done. It doesn't matter what else it breaks, you did your story and are a success.

DevOps - implement often badly tested releases, architecturally inept and with astonishingly bad performance. Blame insufficent monitoring tools for the ensuing network collapse.

It's not just bullshit, it's M&S bullshit...

I would like to read this article but the stupid made up language like "mansplain", "agilesplainers" and "jobby-jobs" put me off so much I had to stop after half a page. I just couldn't handle any more of this garbage.

El Reg please hire adults who speak English to write your articles.

I know my career hasn't been typical...

... but I've read countless versions of this sort of thing over the years and they universally fail to address three major use cases. The first is obvious: safety critical systems. You cannot have "blame free postmortems" when actual postmortems are involved and the people doing the blaming will be the Crown Prosecution Service. This isn't just about aircraft flight control, it applies to far more mundane things like factory conveyor belt systems. Which brings me to the second category:

Systems where the "user" is simply a wetware robot who exists solely to carry out the instructions issued by the software. An early example of this (for me) was a factory making flavours for ice cream: the ingredients had to be mixed in precise quantities with precise timing and moved along a conveyor arrangement. The process could have been fully automated but (back in the '90s) that was far too expensive and retrieving some ingredients from storage wouldn't be possible with a robot even today (without rebuilding the factory).

In such cases "user feedback" usually doesn't matter. The TIM study ("Time in Motion" - remember those?) tells you how long each process takes and the potential for accidents/mistakes is obvious and can be accounted for. Feedback regarding the process itself (how about we add X before Y?) is worthless because the "users" have no conception of what they are actually doing (neither did the programmer - a guy with two PhDs dealt with that). This sort of system gets its feedback from sensors, not users, which brings me to:

Huge numbers of "computing" devices cannot be patched. The software they run when they ship is there for life. You can't (realistically) reflash a microwave or a washing machine and this has spilled over into the sexy new IoT field with the (catastrophic) addition of Agile/SCRUM/"No blame" people let loose on the code. You cannot "iterate" this sort of software. You get it right first time or you're looking at a product recall and/or market vilification.

All these analyses and methodologies seem to start with the assumption that "software" consists of FaceBook and Enterprise CRM systems. It doesn't. In fact such systems are a tiny minority. There are about 50 billion ARM chips out there and about 6 billion smartphones. What are the rest of them doing?

"For me, a 2009 talk by Andrew Clay Shafer codified this thinking right around the time it was codified into DevOps. To a room full of agile lords and ladies, he proposed something wild and crazy: what if you were responsible for how your code ran in production?"

We were doing this over 20 years earlier. I've previously supposed that we were doing DevOps ahead of time. But...

"Systems will go down catastrophically, but you can't simply give up, and punishing people just takes you back to the overly cautious state where software is released infrequently. So, as described by the Google SRE book, you instead celebrate failure"

Thanks for explaining to me that we weren't doing DevOps after all. They didn't go down catastrophically so we didn't have failures to celebrate, only successes to lament.

As a database guy to some extent I've always been puzzled about this salami slicing thing. Do you set up the database by introducing a table at a time or do you set up all the tables empty for the first release and then add the columns one at a time for each subsequent release? And how many release cycles do you expect to have before the developers get anything to develop against. At least as the data volume grows with use each reorg following each sprint's schema change will get longer and longer so eventually there'll be really good failures to celebrate.

Cynical? Moi?

> hoard of "usability" experts?

Grrr. Perhaps you mean horde. And hopefully not whored.

Nice article.

I think that sums up the state of how we got where we are now. Well done.

I'm guessing many of the haters in the comments are 50 somethings, or retired and just don't get all these "new fangled buzzwords" and think that "we did all this devops stuff on mainframes back in the 70s".

As someone who's seen it actually working in practice, I can can tell you that Agile / Scrum / DevOps *can* work... but done wrong it can spit out a crappy product just like any other method. It's not about being like Twitter or Etsy or Facebook or Google... but they DO have some smart people and there's a lot we can learn from them.

The downvote button is the second one along.

Outsourcing doesn't work

because you are mandated to work without a plan.

That either means you have to backpedal and make a plan or be that lucky vendor with a bottomless contract. Can't have it both ways.

In my experience the job always comes with a budget - and the meat (user) in front of you really doesn't care about that - the solution will become a system for one, not necessarily the several hundred in his team.

I suspect most people here are used to dealing with the majority of projects I've dealt with in my career - 2 weeks to 2 years in scale....

but not many dealt with a sub 1 hour build/test/release cycle :-)

Build (compile etc) took about 13 mins, 40 mins for test, then release

I never managed 24 releases in a day that I recall - 23, yes :-) Failures are expected in test, so 48 mins per test cycle (there were additional components such as digital signing needed for a full build that weren't needed for test to start) meant 30 were possible.

My recent teams dealt with Antivirus definitions as well as AV engine functionality - we dealt with Agile by definition, SCRUM was used as applicable (mainly the daily meeting component :-)) waterfall was how we described AV Engine features that took over 30 days to complete!

Oh - and AV researchers wrote engine features as well as vice versa, and the test systems came into the same group, so perhaps can be called DevOps too ;-)

History: Dr Solomon's, McAfee, Microsoft (Defender)

You're suggesting automation doesn't require some level of POC? I suspect most work in an environment where they maintain DEV --> TEST --> STAGE --> PROD. God forbid one just randomly enables some shiny new script directly against prod.