Great, we're going to get DevOps-ed. So, 15 years of planning processes – for the bin?

In large organisations, the question is rarely “what are these newfangled practices and technologies,” but more “how could we actually do them here?” DevOps* has been here nigh on 10 years, and in the past three or so of those, large, normal organisations, like Allstate and Duke, have been learning its mysteries. “I think …

[Anonymous Coward]

as soon as my shop realises

this is more about DEV than OPS the better.

Current obsession with multiple layers of infrastructure and technical tooling being implemented does not change the release cycle, and potentially makes the whole thing even more complex.

The whole point is that infrastructure becomes more of a service regardless of where it is hosted.

They'll get the idea one day...

[Erik4872]

Re: as soon as my shop realises

Yes, the tools is the killer. I'm all for the transition, but the constant swapping out of tools has to slow down if they want anyone other than consultants to develop an aptitude for it. I've actually had people tell me that one tool or another is "old" because it was from 2015.

[Warm Braw]

Where’s the part that ensures the software is actually legal?

Of course it's much easier to deploy changes to software incrementally these days but that's very much a two-edged sword. The software may be modular but the enterprise isn't - it's still one monolithic unit as far as the law and its customers view it. If that "small project" results in a duplicated database of customers that slips below the GDPR radar or whose tool-hire transactions are invisible when marketing decide to institute a loyalty programme then it is the enterprise that takes the hit. That's why those CABs are there and as small projects are just as capable of producing enterprise risk as large ones they won't be going away. "Small batch" may simply be too small given the level of necessary process overhead.

[Andrew Commons]

Re: Where’s the part that ensures the software is actually legal?

I would like to see how this gets implemented in an organisation exposed to the Sarbanes-Oxley Act. I was with an organisation that was exposed to it and had to devise something that clearly showed that everything that hit production was known and approved by management before we got hit by an audit. Being able to reconstruct Production as it was at any specific time was also in the mix. Doing this convincingly in DevOps must be interesting.

[Steve Davies 3]

Re: SOX and DevOps

The typical PHB Nightmare. It means that they actually have to do something for a change instead of issuing Edicts right, left and centre.

[[e-mail]]

Re: Where’s the part that ensures the software is actually legal?

Hi Andrew,

When both your application and infrastructure are built from code and you're using good processes (e.g. git pull requests) with tools like jira and slack you can have a complete digital audit trail from management approval, code review, test coverage and results to release into production with who touched it at each point being automatically recorded.

I reckon compliance is easier in a devops world :-)

[Andrew Commons]

Re: Where’s the part that ensures the software is actually legal?

@Rob@rob

A good tool chain can certainly help. I assume you are doing this with requirements as well and test coverage includes requirements traceability that extends through all work products? In other words end to end with senior management using the tools as well? Tampering with the audit trail not possible, integration between different audit trails all present and correct, and the entire chain future proofed so it can be pulled in 10+ years time?

[Anonymous Coward]

Re: Where’s the part that ensures the software is actually legal?

Our company introduced CAB last year. And in our company, CAB is run by the Infrastructure team. They are not concerned about legality or anything like that - they are simply concerned that if something goes wrong theirs would be the first phone to ring.

So, months of prioritised development effort, all finished, tested, signed off, waiting to go out for a business-critical deployment can be torpedoed because a network guy gets worried about it and blocks the deployment.

This has happened. Expensive (paid-for) marketing campaign ready to drop in the media, and the required change was blocked by somebody who had nothing to do with any of it, in case it caused some site issues.

After that, CAB was scaled back rather dramatically.

[K]

More DevOps nonsence..

I'm not even going to bother commenting on it... oh crap!

[matjaggard]

Re: More DevOps nonsence..

I'd wager that I could guess your age to within 5 years based on your comment.

[Anonymous Coward]

My organisation is currently going Agile(tm)...

...and this is of course akin to asking a fat person to suddenly get off the sofa, put down their KFC bargain bucket and go for a 10k hike. So far the results have been surprisingly amusing, but at some point, we're actually going to have to deliver something. That won't be pretty.

[FozzyBear]

Re: My organisation is currently going Agile(tm)...

My organisation has been trying to understand Agile approach. Now I know how to put this in PHB speak.

[Anonymous Coward]

Oh well

In large organisations, the question is rarely “what are these newfangled practices and technologies,” but more “how could we actually do them here?”

And in successful organisations, the question is “how can we do more of the good stuff that works for us and less of the bad stuff that doesn't?”

Wearing the most expensive trainers in the world won't make you a better runner. Going out and running will.

[Anonymous Coward]

Re: Oh well

"Wearing the most expensive trainers in the world won't make you a better runner."

Totally agree - but it'll make you look like a better runner, and that's more than enough for me!

But then I am morbidly obese. Your milage may vary.

[Anonymous Coward]

Re: Oh well

> But then I am morbidly obese. Your milage may vary.

You would be surprised at the punch some fat fuckers can pull. Never judge too soon. I've been outclimbed and outrunned by bastards twice my size *and* age, and I'm an ex-athlete. :-(

[peterjames]

Re: Oh well

This is like 85% of the organisations out there - essentially uninterested and complacent.

The why of it is clear - but then when people look surprised that china is taking over and democracy going belly up is when it gets entertaining.

It'll be a show to watch in the next 15 years - the superiorly footweared uninterested obese vs the barefoot highly motivated spartans (who on the side run all the nail shops, so you won't be able to set them up with those either).

[batfink]

Usual DevOps bollocks

So, this whole article can be boiled down to "Processes? We don't need no steenkin' processes!". In the author's opinion, the Dev teams should be able to create any old crap as they see fit, without being constrained by architectural reviews or appearing at Change Boards to convince people that they're not going to break something else when introducing their latest wheeze.

The throwaway comments about Ops were telling. No more tickets? Really? How then does the organisation know whether they need more/fewer Ops? It might work ok if they had dedicated Ops for each app I suppose. Nice and efficient...

It's just storing up problems for the future. But that's alright, it's the latest magic bullet, so what could possibly go wrong?

[Anonymous Coward]

Re: Usual DevOps bollocks

You're obviously not on board with the NoOps movement. Get with the times Grandpa, serverless computing Just Works and we don't need any of those pesky systems people anymore. And if you can't do serverless, at least do containers! Every application is a web application, right?

[kryptylomese]

Re: Usual DevOps bollocks

For NoOps to work, Dev's need to think also like Ops and that is not where the glamour is to be found!

[peterjames]

Re: Usual DevOps bollocks

Containerless and codeless too mate - why do we need the code in the first place, that's so 2017...

You know what codeless computing is called?

COTS - your infrastructure.

[matjaggard]

Re: Usual DevOps bollocks

You really don't get DevOps do you? It's not about having no process, it's about AUTOMATING your process. Need to ensure that GDPR is easy to apply, then put an automated system for ensuring consistency in. Step 1 might be to have a manual step in your pipeline if the data stored needs to change - now 80% of your changes don't need to go via any people. Then you look at the next step of automation to remove more human dependencies.

[batfink]

Re: Usual DevOps bollocks

@mat - yep nice theory, and fine for trivial changes. And yes, it can streamline the grunt work.

However, unless you have managed to build your nice expert system, defining all of the interdependencies in your estate, there will be no substitute for the usual bearded 20-years-of-service arsehole pulling you up in the CAB and saying "This Change you're making to system X will bring down system Y because you've not thought it through now run along and try again. And by the way, I see that you ticked the boxes in the approval system to say that your new design complies with the architectural principles, but it doesn't really, does it?".

And good luck building the system that's going to automate all that knowledge and be able to do proper reviews.

[matjaggard]

Re: Usual DevOps bollocks

Reviews are clearly a sensible part of software engineering, but working in an environment where changes to system X can bring system Y down are going to be slow - that's one of the reasons why big organisations need to invest in automation through DevOps, otherwise they will be squashed by smaller competitors who don't have that legacy issue.

[Robo167]

Re: Usual DevOps bollocks

@bat

I think you might have missed a few key points here.

The “goal” is to automate and break out your monolithic applications into smaller, more manageable components. Have API connectors between apps etc.

Application X should never break Y through a change. In fact, application X should raise an error in the connector and Y should continue to work, albeit possibly with some reduced functionality for a time.

Testing still happens before anything is released, it’s just done as soon as the dev is completed so changes are rapid.

So an API not passing the pre-defined tests means it doesn’t go in, and doesn’t break the application.

Anything not caught by tests obviously is an issue - but designing the app to not fall over (through apis etc) means the risk is minimised.

Furthermore, a decent tech stack (git, Jenkins) means you can roll production versions back at the click of a button.

So X-2.0 might break Y, but you can turn it back to X-1.7 at the drop of a hat.

I used to be on the other side of CABbies, and they were 100% political nonsense.

Why do you need a defined meeting about changes? Surely any discussion about X and Y can just happen between the two or more devs who are involved? Else you’re wasting 20/50 minutes of people’s time who only have one relevant part in the CAB.

[Missing Semicolon]

... but how the heck to you budget?

If the actual amount of work and kit only becomes apparent when you're nearly done, how do you price a delivery in the first place?

[Anonymous Coward]

Re: ... but how the heck to you budget?

> how do you price a delivery in the first place?

Optimistically.

[Anonymous Coward]

I'm not a developer but this is a genuine question...

If you don't do a whole lot of of up front analysis and design how do you know that somewhere down the road you won't find you've painted yourself into a corner? That a desperately needed new module requires a ground-up re-design of your iterated wunderkind?

[Erik4872]

Re: I'm not a developer but this is a genuine question...

Thst's the eternal question. It usually gets answered by saying that no one portion of the system is big enough to break anything. In the microservices world, it means the application is broken up into a ton of small components all HTTPing and JSONing their messages around. Even the database is abstracted away so you can swap in a whole new database under the hood.

Where it breaks down is when you start factoring in dependencies, requiring lower latency, etc. All those hops cost time.

[Pascal Monett]

Re: "no one portion of the system is big enough to break anything"

Sure. Go say that to Azure or AWS guys who've repeatedly seen entire continents go TITSUP because of some lone router change or command feedback issue.

DevOps to me seems like a pit where you throw all the tools, throw in the developers, have the band start up some shrieking version of Hells Bells, and hope for the best in the frenetic activity that ensues.

What I want to see is how this new fad will stand up in ten or twenty years' time. And what new fad will have taken the board by then.

[Anonymous Coward]

Re: I'm not a developer but this is a genuine question...

The corner you painted yourself into will be added to the technical debt and NEVER addressed again.

Despite many protests on my previous project that 'It won't work in a million years' the soldiered on and on and on. Thankfully, my contract ended and I declined a renewal.

Six months later and many tens of thousands of Euros later Senior Management realised what had happened and canned the whole thing. Then they started again with the same people in charge. Doh!

[Anonymous Coward]

Re: I'm not a developer but this is a genuine question...

> Six months later and many tens of thousands of Euros later Senior Management realised what had happened and canned the whole thing. Then they started again with the same people in charge. Doh!

Seen that happen too! In a previous life, I got called it to assess a certain system that had been in development hell for years. I spent a few weeks being trained in the system at the development centre and then went out to put it through its paces, even though by that point I knew all too well where the problem was. The night before leaving for the test site I told my client, off the record, that they had a systemic problem in that the development team were incompetent (I do not say this lightly, they actually and objectively lacked all four of skills, experience, qualifications and luck). I went out and tested anyway. When I got back the development team were no more, the parent company having closed down the site.

So far so good. New start.

Except, I get called in to go and assist the new team get started and who were there? The most useless of the useless old team. Turns out that they offered people the option of redundancies or joining the new team, in a different country, with much worse conditions. Of course, only those who were unemployable took the latter option.

Must be about fifteen years now. Last I heard about three years ago they were still trying to get this project out the door.

[Anonymous Coward]

Re: I'm not a developer but this is a genuine question...

> If you don't do a whole lot of of up front analysis and design how do you know that somewhere down the road you won't find you've painted yourself into a corner?

That's where you separate the men from the boys. B-)

[a_yank_lurker]

Re: I'm not a developer but this is a genuine question...

One of the flaws of any system is how to get enough and the right kind of preliminary design data, use cases, etc. collected to get a grip on the size, scope, and complexity of the system. Many IT failures start at this phase as no one made an effort to collect it or make sure what they did have was reasonably relevant. Agile and DevOps try to side step the problem by doing the analysis on the fly as you work on the project. But this fails to understand that one should know why you are doing the project in first place and have a clear idea what a successful project will do.

[Anonymous Coward]

Re: I'm not a developer but this is a genuine question...

You're missing the point. The whole point of DevOps is to shift blame and workload entirely onto the developers.

No specification? Tough, just keep guessing and shoving out updates until your guess equates to the requirement.

No oversight? Well that's your fault, that suddenly was the developers' baby as well.

Blame? Oh that goes squarely into the developers' lap, because if the solids hit the ventilation, management didn't know what the naughty developers were up to.

DevOps is an utter abrogation of management responsibility, and is deliberately engineered so that the developer gets the blame if it goes wrong whilst the management gets the credit if it goes right.

It is a fraud built on a Ponzi scheme anchored in a software South Sea Bubble.

[Erik4872]

The key seems to be doing only what works

It's like ITIL. I've worked in a few places where they've hauled the management consultants in, paid them a shipping container full of money and bought the IT Service Management Platinum Package. Problem is, if you don't adapt ITIL to the organization, it just becomes a ton of paperwork and process that no one wants to follow, and slows down the pace of change. Eventually people get fed up and the situation gets bad.

The same thing is happening again, with fear of missing out stirred into the mix. Everyone wants to be Netflix, Facebook or Google. And they're desperate to do something, anything, right now. What they're failing to realize is that this transition is meant to be adapted as well as adopted. If you have a mission-critical system being supported by 200 different offshore development houses writing code, you really shouldn't trust them to check code directly into production. If you have developers who can't/won't test their code and rely on integration/QA to find their bugs, then they can't "own the problem" the way Google SREs can. Bottom line is that if you want a real transition and not just a tickbox, you have to go slow and fix your cultural issues first. Problem is that tool and consulting vendors don't make money when you do that...they make money scaring people into implementing whatever's hot that month.

[Mike 16]

Re: The key seems to be doing only what works

--- Everyone wants to be Netflix, Facebook or Google. ---

Of course they do, like everybody wants to own a gold mine and nobody wants to be stuck down a mine-shaft 18 hours a day. Netflix/Facebook/Google "customers" (aka "product") may not be so overjoyed.

I will admit that I never did "I.T.", but rather embedded systems, which had the quaint notion that if the software screwed up, the manufacturer of the device that contained it (and whose purchaser often had no idea even had a computer) was on the hook for making it right.

This mania for "move fast and break things" and "change is good" seems to only be from the point of view of the folks cashing in, not on the "customers" (what happened to "stakeholders"?) subjected to "Your inability read your old documents on our shiny new system is not our problem. You should just be happy we gave you a new poop emoji with animating wet-shimmer"

[Anonymous Coward]

Time Scale

So, for DevOps to work you need in-house developers who knows the business, the people and the culture instead of outsourcing and contractors.

[Anonymous Coward]

Re: Time Scale

This is what new adopters totally don't understand. Google et al pay their developers insane amounts of money, only hire the best, and have an insane work culture that basically has them living at work. It doesn't work when you have bargain-basement outsourcers or even low-skilled in house devs writing your software, because the whole idea of DevOps is that the developers have more skin in the game and have to fix operational problems on their own.

[Anonymous Coward]

Re: Time Scale

What you also need is a simple isolated application that is non-critical (no one dies if it screws up) and doesn't need to interact with a lot of other critical systems, and developers of above average competence.

Curiously, replicating the performance of a pile of Google PhDs or university researchers is neither trivial nor inexpensive.

And yes, I did what is now called agile, or RAD, or DevOps developing systems to manage investment strategies for hundreds of billions of dollars... and I've seen how it can fail, as well as succeed, in half a dozen industries.

It is suitable for some purposes, usually user facing relatively undemanding applications, and not at all for mission critical, security critical, reliability critical, highly interconnected (to other critical systems, usually not under your control), safety critical purposes.

It is 'not fit for the purpose' when you are talking Air Traffic Control, SCADA, flight control systems, weapons control systems, emergency communications systems, medical systems, police databases, military command and control, traffic control, railway signalling, systems with mandatory testing and verification regulations, etc. I don't want some enthusiastic, probably relatively inexperienced, DevOps type breaking the control system for the local nuclear reactors, or even the local oil refinery or ambulance/fire/police dispatching system.

We have a cluster of them at my current work, and they are really, really bad about understanding security, infrastructure, standards, troubleshooting in complex environments, and interaction with other systems.

They also have a really high opinion of their own competence and a low one of anyone who actually thinks that Internet and industry standards count for something or who has actually seen some of the other IT 'magic bullets' come and go. Can everyone spell 'Dunning - Kruger'?

[Anonymous Coward]

Re: Time Scale

What's important is having a product architecture you can blog about. :-)

[Anonymous Coward]

Re: Time Scale

"What's important is having a product architecture you can blog about."

Again and again, forever, because it keeps changing?

[peterjames]

Where's the 'PROMOTED CONTENT' tag in front of the title?

Or has El Reg commited to a particular $DEITY and from now intends to preach $DEITY.GetGospel() as its standard?