back to article Culture, schmulture. DevOps, agile need to be software-first again

"The talks get a little repetitive, don't they?" she said as we were walking out of the elevator and through the lobby, escaping the latest two-day DevOpsDays nerd fest. Unable to resist the urge to mansplain, I meekly volunteered that most of the attendees are first-timers, so, you know, maybe it's new to them. Upstairs …

  1. m0rt

    "In the tech industry, we're never really sure which is more important: the tool, or how people use the tool."

    "with many doubts that IT's not up to the task of transforming to the point where they can reliably create, refine, and run software"

    Two statements that stand out, for me at least, why there is still issues with software despite it now becoming (has become?) a mature industry.

    Ultimately it is the users and the consumers, of the software that should be put first. Because the software is pretty much that - the interface between a person and allowable actions, and data stored on objects and/or persons. If you forget that, then you will ultimately build your software in such a way that it will have flow paths that are at odds with what it will be used for.

    Failing often, in my book, is not optional when you are dealing with either a critical system or confidential data. I think Agile gets a bad rap because there are lots of things that get ascribed to Agile, that are not really part of the 'manifesto'. Failing often, for example. The manifesto states Working software...

    Agile got co-opted into a term which gets plastered over any shop's dev process as a way of saying 'me too', similar to the way TWAT™ gets used. It is a way to cover up having to explain what you really mean.

    Anyway, i cringe when I see the term DevOps. I know plenty of Devs who shouldn't be allowed near infrastructure, and plenty of Ops that shouldn't program. There is different set of drivers for both, rightly so. Otherwise, if the be-all and end all was a jack of alltrades, then why do we have security professionals?

    1. hplasm


      "In the tech industry, we're never really sure which is more important: the tool, or how tools use the tool."

    2. Mark 65

      There is a slight rub with putting the "users and consumers" first - they have to be efficient ones. One problem I have encountered time and time again is companies and workers that have inefficient and illogical processes that are an artefact of how things used to be and their mantra of "that's how we've always done things". Their unwillingness to adapt over time leads to software that needs to fit with a bad workflow and then becomes part of the problem rather than the solution. The culture of a company is most often its problem, the software just a manifestation of it.

  2. Aristotles slow and dimwitted horse

    I enjoyed reading this, but...

    As per the title, I enjoyed reading this but like most evangelical treatises about DevOps, Agile and such like, they all seem to refer to some form of utopian dream or of how these "should" work in the workplace rather than providing hard and fast case studies of actual issues and how they were overcome; and when I say that, I mean in relation to the transformative elements of totally unique and individual fully global integrated ERP solutions for example (in which I include legacy, mainframe, web and all the other components and teams that make up that solution) or to a two man business in a shed somewhere, rather than just seeming to expect everyone to be and work like Tw@ter or Facebook.

    I know it sounds dull, as everyone mostly wants to make themselves sound smart by talking in acronym, or focussing on all of the glittering sparkly rocketship technology, but I absolutely support the initial view as to why "culture" is the most oft spoken about piece. This is for a very good reason. Get this wrong and you may as well go and burn all of the cash in a field.

    Where I am currently we're in the early phases of implementing these newer methods such as Agile and DevOps - and I'm more than happy to be supporting this. But to do it the supposed "right" way (and I personally do not believe there is a one size fits all "right" way) despite what all of the pointless books and so called "consultants" say, would mean rebuilding the company organisation, people, hierarchies, systems, processes (Business + IT) and almost everything from scratch. Which just isn't going to happen.

    Each company is unique in the way it works. Whilst the overarching theories about how to "rapidise" stuff is not, how they are implemented most definitely is, and how that "culture" changes (or cannot change) to absorb them is probably the most important aspect of it.

  3. yoganmahew

    New tropes...

    Agile - do away with subject matter expertise and have 'users' (salesmen, not users) talk to college kids about business needs that neither of them understand, regulated processes that neither of them have heard of, and industry standards that neither of them care about.

    Sprint - smash out any old piece of software onto a shared test system and call the story done. It doesn't matter what else it breaks, you did your story and are a success.

    DevOps - implement often badly tested releases, architecturally inept and with astonishingly bad performance. Blame insufficent monitoring tools for the ensuing network collapse.

    It's not just bullshit, it's M&S bullshit...

    1. Bronek Kozicki

      Re: New tropes...

      Agile - do away with the subject matter "experts" and let the developers learn the subject, from the users.

      1. Khaptain Silver badge

        Re: New tropes...

        "do away with the subject matter "experts" and let the developers learn the subject, from the user"

        Isn't that how things actually get none regardless of the methodology ?

      2. yoganmahew

        Re: New tropes...


        "Agile - do away with the subject matter "experts" and let the developers learn the subject, from the users."

        Indeed, but first do away with the existing developers who are the existing SMEs on the business processes that are already in place. Then get a bunch of hand-wavy "heard this cloud thing is great" salesdroids looking at commission in and a bunch of grads who code from StackOverflow examples and watch them make up something that can never work, no matter how much they try and learn from their disasters.

  4. iron Silver badge

    I would like to read this article but the stupid made up language like "mansplain", "agilesplainers" and "jobby-jobs" put me off so much I had to stop after half a page. I just couldn't handle any more of this garbage.

    El Reg please hire adults who speak English to write your articles.

    1. asickness231

      Oh boo, I got a good chuckle from "agile-splaining". Lighten up!

  5. Lysenko

    I know my career hasn't been typical...

    ... but I've read countless versions of this sort of thing over the years and they universally fail to address three major use cases. The first is obvious: safety critical systems. You cannot have "blame free postmortems" when actual postmortems are involved and the people doing the blaming will be the Crown Prosecution Service. This isn't just about aircraft flight control, it applies to far more mundane things like factory conveyor belt systems. Which brings me to the second category:

    Systems where the "user" is simply a wetware robot who exists solely to carry out the instructions issued by the software. An early example of this (for me) was a factory making flavours for ice cream: the ingredients had to be mixed in precise quantities with precise timing and moved along a conveyor arrangement. The process could have been fully automated but (back in the '90s) that was far too expensive and retrieving some ingredients from storage wouldn't be possible with a robot even today (without rebuilding the factory).

    In such cases "user feedback" usually doesn't matter. The TIM study ("Time in Motion" - remember those?) tells you how long each process takes and the potential for accidents/mistakes is obvious and can be accounted for. Feedback regarding the process itself (how about we add X before Y?) is worthless because the "users" have no conception of what they are actually doing (neither did the programmer - a guy with two PhDs dealt with that). This sort of system gets its feedback from sensors, not users, which brings me to:

    Huge numbers of "computing" devices cannot be patched. The software they run when they ship is there for life. You can't (realistically) reflash a microwave or a washing machine and this has spilled over into the sexy new IoT field with the (catastrophic) addition of Agile/SCRUM/"No blame" people let loose on the code. You cannot "iterate" this sort of software. You get it right first time or you're looking at a product recall and/or market vilification.

    All these analyses and methodologies seem to start with the assumption that "software" consists of FaceBook and Enterprise CRM systems. It doesn't. In fact such systems are a tiny minority. There are about 50 billion ARM chips out there and about 6 billion smartphones. What are the rest of them doing?

    1. Doctor Syntax Silver badge

      Re: I know my career hasn't been typical...

      "All these analyses and methodologies seem to start with the assumption that "software" consists of FaceBook and Enterprise CRM systems."

      I'm not sure an Enterprise CRM system would be a good place to start either. First you need a good RDBMS. The good news is that at least you don't have to be Agile about developing that; you can buy it in. Then you need a database schema to sit on it. And once you have that in production you'd better have it right because once it starts to accumulate data it'll be a pig to do a reorg. If you're clever and take the time beforehand you can make your schema general enough and the application configurable enough to set it up for all sorts of different enterprises but ISTM that that's the opposite of Agile.

      1. Lysenko

        Re: I know my career hasn't been typical...

        Ah, but that's not "Agile". To be agile you need to be implementing both the front and back ends at once, which means the front end people mock up data as JSON objects based on their (non-existent) understanding of robust data models. In phase two you encounter the total impedance mismatch between that and 3NF and then have to try to stitch it all together with ORM spaghetti.

        In phase three you realise that you've buried all the business logic in the ORM so you're not really using the RDBMS (triggers, stored procedures, custom data types, views) effectively and have unnecessary raw SQL being concatenated (shudder) and injected client side thereby undermining security and subverting the DB's ability to optimise and pre-compile queries. Then, after a while, you realise that the festering mess of your middleware is so complex and deeply embedded that you can't actually change it without stopping the business entirely, so you're forced to tackle your performance issues with hardware or bizarre workarounds like HHVM.

        As for your second point, "generalising the schema" is de facto denormalization and logically ends with something like Redis. That's fine if ACID and a stable data model don't matter - but (as you obviously know, I'm just venting) FarceSnapTwitGram+ isn't representative of what keeps the economy running. One lost Firebase entry is likely an irrelevance, but one lost (or even "eventually consistent") flight control record could cause chaos with knock on effects across a continent.

        Far too much of DevOps/Agile/SCRUM is essentially borrowed from sewage engineering. You keep the pipeline (of crap) flowing at all costs and try not to think about how bad the contents smell.

        1. Doctor Syntax Silver badge

          Re: I know my career hasn't been typical...

          As for your second point, "generalising the schema" is de facto denormalization and logically ends with something like Redis.

          Not necessarily.

          Take, for instance some process control situation. You're going to take data out of the database, pass it to some machine, take the results of automated inspect and then either pass the job onto the next machine or send it into some rework loop. You can hard code all these rules. Alternatively you can have tables which define the rules so that your code becomes a rules engine. You want to modify the process? Change the rules. You want to reuse the whole thing in a different process? Set up a new database, same schema, different rules.

          Another instance: you have a block of data which describes the job. You never handle individual parts of it, you just bung it out to the machine that does the job. You could "normalise" it - and I've seen this done against my advice - but, in fact you're breaking the normalisation rules because it is, really, a single entity. The right way is to treat it as a blob, store it as a single entity, output it as a single entity. I watched the spec of the job change and more and more tables were added, more and more code written to split the data up, store it and reassemble it.

          The version I did had even more complex demands than changing the spec of a single product over time, it started with multiple products with different specs to which many more were added over time. No problem, the database didn't need to change as all it saw and all it needed to see was a single column to hold the blob in a simple table to keep track of it. Another line of business with a broadly similar requirement? No problem, a few tweaks of code to handle some different T&Cs and we're good to go. We were being agile but I don't think we were necessarily Agile.

          1. Lysenko

            Re: I know my career hasn't been typical...

            but, in fact you're breaking the normalisation rules because it is, really, a single entity

            That's using an RDBMS as a transaction/version controlled document management system. You can do it with a two column table (PK|BLOB), but this is actually a case where Mongo or Redis (or just the file system) might be more effective. Caching web pages is a similar problem domain with well researched solutions.

            If I end up using PostgreSQL to manage two column PK|BLOB tables then I'm usually doing the equivalent of writing a letter in Excel. An application I work on periodically harvests SNMP data which arrives as JSON objects. Those have vendor specific structures and encodings so I shove them in Mongo and then normalise them into PostgreSQL in the background.

            I could just dump them straight into a GUID|JSON Postgres table and process everything internally, but that would mean slower performance and less resilience to network outages. I could also leave the data in Mongo and analyse it in situ (through an ORM!?), but that would just mean I'm an idiot ;)

  6. Doctor Syntax Silver badge

    "For me, a 2009 talk by Andrew Clay Shafer codified this thinking right around the time it was codified into DevOps. To a room full of agile lords and ladies, he proposed something wild and crazy: what if you were responsible for how your code ran in production?"

    We were doing this over 20 years earlier. I've previously supposed that we were doing DevOps ahead of time. But...

    "Systems will go down catastrophically, but you can't simply give up, and punishing people just takes you back to the overly cautious state where software is released infrequently. So, as described by the Google SRE book, you instead celebrate failure"

    Thanks for explaining to me that we weren't doing DevOps after all. They didn't go down catastrophically so we didn't have failures to celebrate, only successes to lament.

    As a database guy to some extent I've always been puzzled about this salami slicing thing. Do you set up the database by introducing a table at a time or do you set up all the tables empty for the first release and then add the columns one at a time for each subsequent release? And how many release cycles do you expect to have before the developers get anything to develop against. At least as the data volume grows with use each reorg following each sprint's schema change will get longer and longer so eventually there'll be really good failures to celebrate.

    Cynical? Moi?

  7. nijam Silver badge

    > hoard of "usability" experts?

    Grrr. Perhaps you mean horde. And hopefully not whored.

    1. Doctor Syntax Silver badge

      "Perhaps you mean horde."

      I think hoards was better. After all, hoards are often buried or at least hidden out of the way.

    2. DainB Bronze badge

      herd, obviously

  8. Steve Button Silver badge

    Nice article.

    I think that sums up the state of how we got where we are now. Well done.

    I'm guessing many of the haters in the comments are 50 somethings, or retired and just don't get all these "new fangled buzzwords" and think that "we did all this devops stuff on mainframes back in the 70s".

    As someone who's seen it actually working in practice, I can can tell you that Agile / Scrum / DevOps *can* work... but done wrong it can spit out a crappy product just like any other method. It's not about being like Twitter or Etsy or Facebook or Google... but they DO have some smart people and there's a lot we can learn from them.

    The downvote button is the second one along.

    1. Lysenko

      Re: Nice article.

      As someone who's seen it actually working in practice, I can can tell you that Agile / Scrum / DevOps *can* work...

      ... and I can tell you that PRINCE2 and SSADM *can* work. Smart people were/are involved in those too. Does that mean they should be evangelised as generally applicable innovations that will transform the the face of software engineering? No. Let me, correct that: Hell No!.

      Building Bezos' Bookshop using PRINCE would be as insane as building a rail signalling control system with SCRUM ... the (worrying) difference is that only PRINCE seems to realise that.

      NB: Retired, no, age, wrong, mainframes, never touched one.

      1. Khaptain Silver badge

        Re: Nice article.

        Prince2 and SSADM:

        Strangely enough these are the only two methodologies that I have studied/certified for.

        In a small shop neither is truly applicable unless they are severely chopped up into a fraction of what they represent.

        I have never developed in a large shop so I can't say if they are actually usefull or not..

    2. lleres

      Re: Nice article.

      No, wrong on all counts. The push back is from seeing these things being applied, or attempts thereof, to everything regardless of the suitability of those tools to the task.

      "DevOps" is slang for "we dev, you ops". For developers it means they get to write code for ops to use. For ops it means they get to write code for devs to use. See the problem here?

      "Agile" is slang for "let's spend 2 weeks doing multiple POCs that we know right now are not a good fit for what we want but will do anyway because Agile".

      The fist time i heard the term "scrum master" I thought it was a joke. Then I saw actual job titles with that term. I still wonder what these people do, after years participating in scrums.

      I just boil it down to "Automation". If it saves time, it's worth doing. Fucking about doing POCs and scrums while pretending to be working is not being "agile", it's just procrastination.


    3. Anonymous Coward
      Anonymous Coward

      Re: Nice article.

      I've watched the same team succeed, fail or somehow manage both simultaneously with the same Agile/Scrum workflow! It was impressive seeing the teams productivity, not so impressive when all that extra work just got thrown away over and over again.

      I'm used to killing a lot of experimental code on the way to a solution but these uberorganised workflows almost seem designed to output finished product only by accident.

    4. Doctor Syntax Silver badge

      Re: Nice article.

      I'm guessing many of the haters in the comments are 50 somethings, or retired and just don't get all these "new fangled buzzwords" and think that "we did all this devops stuff on mainframes back in the 70s".

      I'm sure we're mostly in those groups. I certainly am. Mainframe? Yes, I learned FORTRAN on a University mainframe back about 1970; haven't use one since.

      Buzzwords? Let's be careful here. We've seen a lot of technology change and it's all had its own terminology (and I'm convinced more changes of terminology than were strictly necessary - OO didn't really need to start from scratch given that it essentially added to existing concepts). But basically it's just the names for actual stuff, stuff that we've grabbed with both hands when it offered something useful. Microprocessors from 8-bit upwards? Great. 8" floppies? 5 3/4"? 3 1/2"? QIC? CD? DVD? MFM? ATA? SATA? USB? RDBMS? Yup, all good stuff, all grist to the mill. All mastered and taken to heart. Even just listing a few makes me realize, yet again, how much has changed and how we've all changed with it.

      Management buzzwords, methodologies and the search for the Silver Bullet, that's a different matter. We've seen a lot of those go past us as well. The real difference that age brings isn't being unable to handle the buzzwords, it's that we've seen the same things go past again and again, just with different labels on them and the same mixture of good and bad results. The reason management prefers young coders to the old and grizzled is that we're resistant to bullshitting which is what they do best, and often the only thing they do at all. What we've learned is that there's no Silver Bullet, the best approach is to look at what you need to do, choose the approach that seems to fit the job in hand best and ignore the labels.

      You'll eventually look back and say the same thing. The only difference will be your list of technologies.

    5. albaleo

      Re: Nice article.

      "I can can tell you that Agile / Scrum / DevOps *can* work"

      I was reminded of a couple of stories. (I'm past my fifties, but only entered software development seriously after I was 40.)

      One story concerns the idea that "anything can work". It was prompted by a senior engineer at a Japanese steel company in the early 80s. At the time, Total Quality Control was the thing (Deming Prize, etc.). He frequently had to entertain visiting groups from the United States who wanted to pin down the the proper way to conduct TQC. He was badgered with questions about which method of TQC was best. He told me that it didn't matter which method you chose, just so long as you had a method, generally stuck to it, but had some notion of your objectives so you could break the rules when necessary.

      Another story concerns the teaching profession, again in the 80s, when the notion of "Student Centered Learning" was introduced at a conference I attended. It seemed to many attendees that various "youngsters" were trying to formalize something that was taken as normal good practice among experienced teachers. And there was that dread feeling that by giving it capital letters , it was already doomed. (Another poster hinted at a difference between agile and Agile.)

      I've enjoyed reading the comments here. I do wonder sometimes whether the different opinions are based around the different tasks we do and also our various backgrounds. Creating reporting applications for educational achievement and creating controllers for industrial machinery seem worlds apart to me.

    6. Anonymous Coward
      Anonymous Coward

      Re: Nice article.

      "I think that sums up the state of how we got where we are now. Well done.

      I'm guessing many of the haters in the comments are 50 somethings, or retired and just don't get all these "new fangled buzzwords" and think that "we did all this devops stuff on mainframes back in the 70s"."



      At least, I did all this devops stuff on mainframes in the early 70s.

      Of course we didn't break things, as there were hundreds of billions of dollars being steered by our system, and we didn't let end tell us how to design it. They really liked it, though, enough to trust a lot of money to it, back when a billion dollars was worth about 10 times what it is now.

  9. Anonymous Coward
    Anonymous Coward

    Outsourcing doesn't work

    because you are mandated to work without a plan.

    That either means you have to backpedal and make a plan or be that lucky vendor with a bottomless contract. Can't have it both ways.

    In my experience the job always comes with a budget - and the meat (user) in front of you really doesn't care about that - the solution will become a system for one, not necessarily the several hundred in his team.

  10. DanW

    I suspect most people here are used to dealing with the majority of projects I've dealt with in my career - 2 weeks to 2 years in scale....

    but not many dealt with a sub 1 hour build/test/release cycle :-)

    Build (compile etc) took about 13 mins, 40 mins for test, then release

    I never managed 24 releases in a day that I recall - 23, yes :-) Failures are expected in test, so 48 mins per test cycle (there were additional components such as digital signing needed for a full build that weren't needed for test to start) meant 30 were possible.

    My recent teams dealt with Antivirus definitions as well as AV engine functionality - we dealt with Agile by definition, SCRUM was used as applicable (mainly the daily meeting component :-)) waterfall was how we described AV Engine features that took over 30 days to complete!

    Oh - and AV researchers wrote engine features as well as vice versa, and the test systems came into the same group, so perhaps can be called DevOps too ;-)

    History: Dr Solomon's, McAfee, Microsoft (Defender)

  11. Malignant_Narcissism

    You're suggesting automation doesn't require some level of POC? I suspect most work in an environment where they maintain DEV --> TEST --> STAGE --> PROD. God forbid one just randomly enables some shiny new script directly against prod.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like