back to article Refactoring whizz: Good software shouldn't cost the earth – it's actually cheaper to build

ThoughtWorks chief scientist Martin Fowler has written about the curious inverse relationship between quality and cost in the field of software development. The user cannot distinguish between good or bad internal design simply by observing the user interface or the features, he observed. This means that poor-quality software …

  1. alain williams Silver badge

    This is nothing new

    I have been writing programs for 40 years, I remember reading something early on about the importance of design: a unit of time spent here would be repaid by 10 less units in coding. Similarly a unit spent testing would be rewarded by 10 fewer units of support once the software reached the users.

    Much blame lies with managers who want the kudos of a project quickly delivered but then blame their staff once problems emerge.

    1. JCB

      Not My Budget

      You are missing the "not my budget" problem. Develop the application within budget and working well enough, then everyone is happy. Of course we all know that any project of any scale will have bugs. But these bugs don't show up until after delivery. Bugs get reported and possibly come back to the same manager as supervised the initial development, but he is still happy as the hours fixing it will be charged to a warranty budget. Staff are kept in work between projects and the department has additional warranty money coming in to cover it. Manager and staff remain happy.

      1. Doctor Syntax Silver badge

        Re: Not My Budget

        Budgets only work to manage costs when there's somebody keeping an eye out on the overall budget and, in fact, on other stuff. I've seen maintenance type done n stupidly bad order, presumably by separate budget holders not coordinating so one operation comes along and craps on the new paint job that was just finished.

    2. Ian Michael Gumby
      Boffin

      @Alain Williams ... Re: This is nothing new

      Yes, I agree.

      However its not just the managers not allowing for enough time.

      Its also the skill level of the developers.

      A senior level developer can crank out solid code faster than a junior developer.

      The quality of code and thought is also a key difference.

      Yes, I know that this is pretty obvious.

      However... What isn't obvious is that many have the title of senior developers who still write crappy code. Too many developers write code that doesn't scale and just barely works. That's the problem.... and I think that's the point of the article.

    3. Blank Reg

      Re: This is nothing new

      A problem that I've seen is people spending months on design when I could get v1 up and running in weeks. And at the end of that prolonged design phase, there are still too many issues because you only really understand a problem after you've solved it, even if you solved it with the worst code on earth.

      So my preferred approach is to do minimal design up front, get something working so that you understand the problem, then throw that away and do it right.

      1. Anonymous Coward
        Anonymous Coward

        Re: This is nothing new

        You are Bill Gates. The trouble is, they've been trying to "do it right" so many times I've lost count.

  2. Roger Greenwood

    If you want to go fast . . .

    then Fred Brooks has a suggestion for you:- plan to throw one away, you will anyway.

    In the real world of hard stuff we call it prototyping and you learn a lot from the mistakes, but at some point you need to start again. Whether you do or not will be a political/ego decision not a technical one. The costs accrue accordingly.

    1. Daniel von Asmuth

      You must always go fast...

      That is better. Real world programmers have deadlines to meet, but good quality software takes time, so they build a quick demo. Then they can make a good design and implement it properly, eliminating the need for expensive fixes after release.

      1. Tomato42

        Re: You must always go fast...

        except a PHB comes in after a demo is ready and tells them that since they have this thing ready, they can work on this new project

        1. Doctor Syntax Silver badge

          Re: You must always go fast...

          Or complains that the demo doesn't do this or that right so the team couldn't have been up to the job. I had a friend who suffered that one repeatedly - manglement couldn't get their heads round the fact that the demos were information gathering exercises.

    2. J.G.Harston Silver badge

      Re: If you want to go fast . . .

      Ah, that explains Britain's problem with the EU. We've been running on the development version for 40 years and bolting on random additional "oo! shiny" features.

    3. Carlie J. Coats, Jr.

      Re: If you want to go fast . . .

      What he actually said is--throw *two* away: "If you have a new problem that needs solving, then do the best job you can of analysis, then design, then implementation. And you will have something that *almost* works. Throw it away and take a two month vacation. Come back and again do the best job you can of analysis, then design, then implementation. You will have something that solves the problem you were originally trying to solve. But it is brittle, and lacking adaptability. The long-run worst thing you can do is to use it. Throw it away and take another two month vacation. Come back and again do the best job you can of analysis, then design, then implementation. This time you will understand the "problem behind the problem" and will come up with an implementation that is elegant, extensible, and robust...":

      1. Charles 9

        Re: If you want to go fast . . .

        Trouble is, who has that time when they normally have too short a deadline and a DIE directive?

  3. knarf

    More babblings from Mr Fowler

    See he's running with another Trend

    1. PghMike

      Re: More babblings from Mr Fowler

      No, these trends he’s talking about are over 50 years old. Fred brooks wrote about them when writing about what went wrong writing software for the ibm 360, in the 1960s

      1. richardcox13

        Re: More babblings from Mr Fowler

        And Fowler has been talking about than for over two decades.

        That the message that keeping the code clean and maintained saves money hasn't got through to many is the sad part. Too tied to detailed plans that don't survive the few days coding.

        1. Ian Michael Gumby
          Boffin

          @Richard Cox ...Re: More babblings from Mr Fowler

          I think the issue is that people want code that works. At least works well enough.

          They don't care about maintenance of the code because they believe that by the time you need to upgrade the code, it will be obsolete. Who cares about the legacy COBOL code that's been running for 40 years and is business critical code?

          I mean suppose you have a software project and doing it right costs $250,000 USD. Compare that to using offshore resources that can barely code but cost a fraction. So you can save $100,000 USD for code that barely works but checks all the boxes.

          Clearly we would choose to spend the $250K USD. But the pointy haired manager wouldn't because he'll use that savings to justify his bonus. And when things fail... he'll have someone else to blame.

  4. Buzzword

    Ceremony over results

    Alas, Fowler's message often gets corrupted when implemented. For example, we spend a lot of time writing unit tests which raise the Code Coverage metric and make managers happy, but which don't really test anything useful. The least reliable parts of any system are the pieces of string connecting the black boxes, not the black boxes themselves. At scrum planning sessions, we meticulously estimate every task down to the nearest hour; only to find those estimates blown out of the water within minutes of starting the task. Similarly, at scrum review we obsess over why the burndown chart has a long plateau followed by a steep drop on the last day; but since fixing that would require re-structuring the team, it never improves.

    I'm a big fan of Fowler's work; I'd just like to see better implementation guidance.

    1. Charlie Clark Silver badge

      Re: Ceremony over results

      Unit tests and coverage are great but they are not goals in themselves. While they do test the functionality of bits of code, they say nothing about design or suitabiltiy. My experience is that they are really useful for maintenance and refactoring. And, of course, they get people used to the idea of testing…

    2. Gonzo wizard

      Re: Ceremony over results

      "We spend a lot of time writing unit tests which raise the Code Coverage metric and make managers happy, but which don't really test anything useful": You're writing the wrong tests if that's all they achieve. Unit testing is an important part of demonstrating that the code does what is intended.

      "The least reliable parts of any system are the pieces of string connecting the black boxes": That may be true but doesn't detract from the problem that poorly tested software will be an equally big headache. In any event the user probably can't see the difference anyway, and will simply say that "the system didn't work".

      "we meticulously estimate every task down to the nearest hour": Oh dear. In all the agile teams I've worked on we rarely went lower than T-shirt sizing or units of half a day. Anything more detailed is a waste of time and smacks of waterfall disguised as agile.

      "I'd just like to see better implementation guidance": Start with "ReWork: Change the Way You Work Forever" which talks about how to make agile work for you. It's up to the people in a team to make changes. If your team isn't allowed to change the way it works, leave.

    3. Anonymous Coward
      Anonymous Coward

      Re: Ceremony over results

      "At scrum planning sessions, we meticulously estimate every task down to the nearest hour"

      If you're doing that, then you are approaching your planning completely wrong - for exactly the reasons you just laid out. Time-based estimation is almost always wrong, which is why abstract story points based on estimated relative 'difficulty' of each task (eg T-shirt sizes) should be used for sprint planning.

      Over time, you are then able to refine how many points of relative difficulty your team is likely to be able to deliver a sprint based actual vs planned story points, and plan each subsequent sprint accordingly.

    4. richardcox13

      Re: Ceremony over results

      > For example, we spend a lot of time writing unit tests which raise the Code Coverage metric and make managers happy

      Goodhart's law applies.

    5. a_yank_lurker

      Re: Ceremony over results

      Your complaint echoes Uncle Bob Martin's complaints about 'Agile'. Most of what Fowler is talking about is not really procedural but attitude. Do you take the time upfront to get the design, specs, etc right? Are you regularly talking to key stakeholders? Then do you take the time to write and test quality code? These are more attitude than any specific methodology.

  5. Denarius

    Nothing new ?

    Alain, I wish I could agree. IMNSHO, after working for a series of outsourcerers over the decades, is that the immediate PHB does indeed behave as you suggest. However the beancounter and management droids a layer or 3 above want slow crappy code. Much more revenue is drawn from suckers customers who persist in throwing good money after bad. The old sunk capital problem. Perhaps the mantra, " fail fast and fail often" is not so much for code cutters as their firms customers.

    As for articles main premise, I fully agree. Taking the extra time to work out a modular structure pays dividends when releasing updates. Aside from that, the challenge in producing elegant code that works efficiently and well is a buzz for me. Unfortunately there are many code cutters who don't upgrade their code quality(ie competence with language of choice) who mistake elegance for difficulty. ITIRC Dennis Ritchie promulgating a great attitude with his comment about the first thing he did when designing code was push the keyboard away to make room for the paper and pencil.

    1. Anonymous Coward
      Anonymous Coward

      Re: Nothing new ?

      I work in IT for a large company but not as a programmer. The comment below is speculation on my part.

      "...and costs the customer money."" is the whole point of programming shops but only within what a customer would find acceptable.

      Many programming shops will have a huge amount of code that they can reflect upon.

      What of that code was well written?

      What pieces of code were crap?

      What was the cost to the shop of each of those qualities of code?

      What did the programming shop get to charge the customer for fixes versus one time charge for programming it correctly the first time?

      Many customers have no way to evaluate the quality of a product they are receiving and if their new software doesn't work as well as they expected the programming shop can probably suggest that it was built to the customers specifications without any worry that the customer can prove there was a better way and that the code they received is crap.

      Charging for changes is a huge money maker.

      1. Giovani Tapini

        Re: Nothing new ?

        I hear you, but the impact of poor code is far more manageable than the impact of poor thinking through as to how the application should be structured for ongoing feature addition.

      2. Ken Moorhouse Silver badge

        Re: no way to evaluate the quality of a product they are receiving

        One way is to see how long a requested modification takes to roll-out satisfactorily.

        My approach is to try to take into account all the possibilities at the design stage. I really hate having to backtrack to incorporate a new feature. Having said that, backtracking is far superior to "bolting-on" a new feature.

        (I still remember the day I sauntered up to the counter at Dillons to pay for Fowler's "Refactoring". Some guy in the queue gave me the thumbs-up and said "good choice mate").

      3. Doctor Syntax Silver badge

        Re: Nothing new ?

        "Charging for changes is a huge money maker."

        And those changes being harder than they should to implement is a great money loser.

        Rule 1. Development is the process of launching a product into maintenance.

        Rule 2. Maintenance is the longest part of the product life-cycle unless the product was canned PDQ.

        Corollary: Good development prepares the way for easy maintenance.

        Rule 3. Forget the "You ain't gonna need it" stuff. You will. Sod's law variation no 286 says that if you don't have it you're going to need it even sooner.

        Rule 4. Think of the most general problem you can of which the immediate requirement is an example and solve that problem. You might find it's no more difficult doing it that way; if the additional requirements come in while you're working and you were already meeting them it'll actually be easier than chopping and changing.

        Rule 5. If you tailor your program to fit the requirements exactly the requirements will become your program's limitations.

        Rule 6. Your program's limitations may well become its bugs.

  6. Caver_Dave Silver badge
    Unhappy

    Costs of fixing bugs

    The worst I came across was an organisation where code was written by one team, a second team tested it and then a third team fixed the bugs identified by testing or the users. That's fine, until manglement assign different cost centres to each team and then used metrics in completely inappropriate ways.

    The code team were heaped with praise and bonuses for producing lots of eLOC.

    The support team were regularly beaten up about the time it took to fix bugs, which was in a very large part because they had to fix the code teams terrible "write-only" code, without design documentation, intended algorythms or comments.

    The number of bugs/eLOC was not a metric manglement used!

  7. Will Godfrey Silver badge
    Unhappy

    A voice crying in the wilderness

    See title.

  8. Doctor Syntax Silver badge

    "As they do this, their change failure rate is significantly lower than slower organisations [and] they recover from errors much more quickly."

    Unless, of course, the slight error managed to corrupt the live data. Or allow a breach that's been exploited and you have to have a conversation with the ICO. Or something equally nasty.

  9. PghMike

    Plan to throw one away ...

    ... you will anyway

    Mythical man month iirc

  10. Frederic Bloggs

    Writing for change

    The real problem is that nearly no-one writes code that is designed (from the very beginning) to be changed. The fact is that all code will be altered and also added to. Very few people design their systems (never mind programs) with this in mind. Why? Because it's hard - at least until one has done it a couple of times.

  11. Anonymous Coward
    Anonymous Coward

    SW .EQ. intellectual exercise .NNE. Intelligent

    There's an unwarranted assumption that coders are somehow inherently superior 'intellectual creatures' because creating software is indisputably an intellectual exercise (as compared to manual labour like shovelling dirt).

    But it's worth noting that Sudoku puzzles are also an intellectual exercise, but it clearly does not necessarily follow that Sudoku fanatics are highly intelligent There's more-obviously no connection between 'intellectual' activities and 'intellectual' intelligence.

    When I come to power, coders will be forced to use much larger keyboards with very heavy springs, to help instill in them a realization that their labours in fact include typing.

    This may help to make obvious that their 'typing in code' job function does not necessarily support any particular assumption about their own intellectual horsepower.

    This change would make it slightly easier to manage them and their tasks.

    1. A.P. Veening Silver badge

      Re: SW .EQ. intellectual exercise .NNE. Intelligent

      This change would make it slightly easier to manage them and their tasks.

      Right, for all of the half minute it takes to use those larger and thus heavier keyboards to break the skulls of managers suicidal enough to give them those keyboards.

    2. Philip Stott

      Re: SW .EQ. intellectual exercise .NNE. Intelligent

      No inferiority complex to be seen here. Move along people.

    3. Doctor Syntax Silver badge

      Re: SW .EQ. intellectual exercise .NNE. Intelligent

      "There's an unwarranted assumption that coders are somehow inherently superior 'intellectual creatures' "

      Certainly not inherently. I can think of a few counter-examples.

    4. Anonymous Coward
      Anonymous Coward

      Confidence to Competence Ratio

      The point here is about the ratio of Confidence to Competence.

      Too many code typists are so full of themselves, gleefully exercising their buzzing-with-excitement brains all day, never realizing that they forgot to take any formal training on subjects such as Algorithm Design or Data Structures.

      They think Pascal is a unit of air pressure. They're blissfully unaware of fully 90% of the entire field of Computer Science.

      Their Confidence to Competence ratio is dangerously over unity, like 1.5.

      This precise factor is at least fully half the reason why so many software projects become fiascos. It's why self driving cars drive straight into obstacles. It's why space probes crash into Mars. It's why big rockets explode. It's increasingly why airplanes crash.

      A good project manager needs to 'read the Riot Act' to their team of coders at the outset. Hammer their confidence level down to below the level of their competence.

      For whatever reason, this whole thing is much less of an issue with other disciplines. Mechanical and Electrical engineers are naturally cautious, full of self doubt, up all night worrying. Confidence to Competence ratio around 0.7.

      1. Glen 1

        Re: Confidence to Competence Ratio

        "They think Pascal is a unit of air pressure"

        It is.

        Slightly more seriously, an obsolete language rarely used in anger this century is not on my list of demands from employees. Sure, Pascal was the Python of it's day, but that day was 30 years ago.

        The anticipated retort is "It's not about the language, it's about learning/understanding blah blah" to which I reply:

        Then why the dig about Pascal if it's not about the language?

        1. Anonymous Coward
          Anonymous Coward

          Re: Confidence to Competence Ratio

          He didn't dig at Pascal, he was pointing out that many programmers these days seem to avoid the whole learning curve of using different structured languages in an educational context.

          1. Glen 1

            Re: Confidence to Competence Ratio

            They were not digging *at* pascal, but at people who are not familiar with pascal.

            "Too many code typists are so full of themselves, gleefully exercising their buzzing-with-excitement brains all day, never realizing that they forgot to take any formal training on subjects such as Algorithm Design or Data Structures."

            I take the point about lack of broadness in people's knowledge, but let's face it, when was the last time a web dev needed to know big O notation? Sorting algorithms beyond data_structure.sort()?

            All the interesting problems have either been solved or are being worked on by Real Programmers™ . All the nuances are hidden behind library function calls. There is money to be made in the harder maths side of things, but thats called 'data science' or 'research'.

            The jobbing programmer making yet another CRUD app might well the know the bullshit interview question on how to swap 2 variables without using a third, but have the common sense never to use it.

            You see it a lot of the Real Programmers™ bullshit in books and academic papers where the maths is the point, not the actual implementation (which is all a programmer will care about). The arena where single letter variable names - with superscripts and subscripts - are considered essential. The sheer unreadability of it seems targeted to exclude rather than educate.

            Example:

            What is the sum total of the contents of an array?

            Not-Code™:

            For each element of array, add contents to running total.

            Maths speak:

            Sigma something something McSquiggle

            TL;DR Theres a difference between the 'if all you have is a hammer' saying and not hiring someone because they don't know web assembly when the job spec is for angular.

        2. Doctor Syntax Silver badge

          Re: Confidence to Competence Ratio

          "Slightly more seriously, an obsolete language rarely used in anger this century is not on my list of demands from employees."

          When I retired - in this century - my last client had been a Delphi shop. The last I saw of them they were recruiting more and more people on a project which seemed to be getting rapidly out of control to re-write their flagship product in C#. That business no longer exists.

      2. Robert Forsyth

        Re: Confidence to Competence Ratio

        Why not just employ typist, instead of expensive programmers?

        Or just have a motor and a cam bash those keys?

        In your last paragraph, have you discovered the reason for the decline in manufacture?

        It is software?

        When a probe crashes into a planet, because the sensor has been fitted upside down.

        When insulating foam comes loose and knocks some thermal tiles off a wing.

        When the angle of attack sensor freezes up.

  12. Anonymous Coward
    Anonymous Coward

    Lession lernt from HP

    When touring a HP establishment many years ago I asked about quality and bugs during the intial producting phases -"Oh that's easy we put the R & D guys on the production run - they get all the complaints and demand for the fixes right there in a pressured environment - its a lession they learn very quickly about quality"

    If you want to improve software quality put the programers on the help desk - especailly if you have customers with 24x7 contracts. Poor code means poor sleep.

    1. Dan 55 Silver badge

      Re: Lession lernt from HP

      Which means worse code.

      Firefighting on the helldesk and being responsible for the new code at the same time just doesn't work and manglement just use it as an excuse to cut back on QA anyway.

      Try instead: Dev, QA, and acceptance environments which are similar to production (the closer to production, the more similar) and a QA team which actually know the systems and know that a change here means testing those things over there as well, instead of just repeating the unit tests and rubber stamping them because otherwise you might as well just fix code in production.

      1. Doctor Syntax Silver badge

        Re: Lession lernt from HP

        "Firefighting on the helldesk and being responsible for the new code at the same time just doesn't work"

        I read nothing about "at the same time" in the OP.

        1. Dan 55 Silver badge

          Re: Lession lernt from HP

          I must have based my reply on my own experience...

          But are there companies that would take developers completely off development and put them on bug fixing on the initial production run just in case something happens?

          1. Doctor Syntax Silver badge

            Re: Lession lernt from HP

            There's a lot of benefit in getting to know your users. I was fortunate in that I worked largely with in-house users so it was possible to walk over and talk to the actual users as well as their managers. That way you learn what's actually important and what's not.

            1. Dan 55 Silver badge

              Re: Lession lernt from HP

              There's no way I'm breaking through the silo wall to talk to the end users. Too many specs and contracts in the way. Well, too much middle manglement which produces specs and contracts.

  13. LordHighFixer

    When I learned to code

    I was taught that 100% of writing code was only 10% of the work, the other 90% was in code cleanup and bug fixing, BEFORE you released the code to test. I guess someone forgot that along the way.

    1. Anonymous Coward
      Anonymous Coward

      Re: When I learned to code

      "...other 90% was in code cleanup and bug fixing..."

      You're doing it wrong.

      It's supposed to be 95% advanced planning, design (data structures, algorithms, GUIs), and then maybe 5% of typing in some code to implement.

      Once upon a time, I spent several weeks on the first step (the 95%), then I stayed over night (just one night) mindlessly creating and typing in about 30-feet of code.

      Because of this process, the code was bug-free and ready to go the next morning. Never needed touching again.

      The problem with your approach is that there's always one more bug. It'll take years to asymtopically approach perfection.

      1. Anonymous Coward
        Anonymous Coward

        Re: When I learned to code

        Many years ago I had to write a control program in assembler. I did actually do all the programming properly as you suggest. It took me several weeks. On Monday I started to code. On Tuesday I managed to wipe out the code I had written so far - back in the days of floppies, the backup got corrupted. It took less than a day to reconstruct due to having the design to work from.

        On Friday afternoon I suddenly realised that I had three blocks of code; one initialisation, one interrupt routine, one interface. And that was it; burn the ROM and go. And it worked first time.

        But I still pretended to spend most of the next week testing because I didn't want the boss getting ideas.

  14. Anonymous Coward
    Anonymous Coward

    And what about the proliferation of languages

    It seems that every time someone has a problem that has been solved before, they decided to invent a new language, citing some minor "improvement" that make the new one more "productive" meaning more idioms to learn and more gotchas.

    The single biggest abomination must surely be javascript. Wasn't it obvious that mozilla was playing a joke? Any person in the mid 90's would have known this, seeing as there was no java whatsoever in there.

    Having said that there is also this mixing of data with code. Processors are not like that. They know about data and code - and they are separate. It seems that this OO thing likes treating one equal to the other. Needless to say the outlook doesn't look good.

    1. Dan 55 Silver badge

      Re: And what about the proliferation of languages

      As Churchill might have said, "OO is the worst form of programming, except for all the others".

    2. holmegm

      Re: And what about the proliferation of languages

      Wasn't it obvious that mozilla was playing a joke? Any person in the mid 90's would have known this, seeing as there was no java whatsoever in there.

      I believe the original idea was using it to manipulate Java applets ... so, you know, "scripting Java".

    3. Doctor Syntax Silver badge

      Re: And what about the proliferation of languages

      "They know about data and code - and they are separate. It seems that this OO thing likes treating one equal to the other. "

      Not in any implementation I've seen.

      You program in C? If so you'll know what a struct is.

      You define your struct. Somewhere, possibly somewhere else, you write the code that uses the struct. The code will have statements to allocate one or more blocks of struct-sized memory and pass pointers to other code that twiddles bits inside those blocks. You might accidentally pass a pointer to some other struct - shit happens. When you've finished with a block, if you remember, you'll release the block. When you compile it binary code goes into your library or program file. When the code executes and gets to the part that allocates memory for the struct that allocation takes place in the data space. As you say, separate.

      Now take the OO case. You define a data structure something like a struct but in the same section of code you provide the code that operates on that data structure including code to allocate and release space for the data structure. You compile it and binary code goes into the library or program file and when you execute the statement to allocate space for an object it goes into the data space. There really isn't much difference except the terms used and the fact that it's a bit more difficult to accidentally point the wrong data structure to code.

      OK there are some extra bits and pieces such as inheritance but in regard to keeping instructions and data separate any decent OO implementation will do the same thing as your preferred non-OO system.

      OO is really an extension to the way we did things for years and the real problem is that its inventors had to follow the industry standard and invent an entirely new vocabulary and make it into a religion, then add a dressing of hype. It took me years to see through that and when I did I realised it was just an improved version of the familiar.

    4. Robert Forsyth

      Re: And what about the proliferation of languages

      At the level below the program, the program is data.

      A SQL or BASIC program is data to a SQL or BASIC interpreter.

      A C program is data to a C compiler.

    5. Anonymous Coward
      Anonymous Coward

      Re: And what about the proliferation of languages

      Harvard architecture processors know about data and code but they are quite unsuitable for general purpose use and the workarounds to load addition code are clunky.

      Von Neumann architecture computers don't really know about the difference, and the workarounds are also clunky. Many vulnerabilities can be traced back to data being made to execute.

      I'm not sure what your point is.

  15. Gonzo wizard

    Cruft

    Cruft = Software Entropy and if done properly only really starts to happen once something is live. At that point your freedom to refactor starts to be restricted as you have to consider what level of risk (change) can be tolerated whilst still providing software that is stable and fit for purpose.

    1. Anonymous Coward
      Anonymous Coward

      Re: Cruft

      If cruft really was software entropy, you would be able to reduce it simply by compressing it isothermally.

      1. Ragnvald

        Re: Cruft

        That would be called 'refactoring' in software neo-speak.

        1. Ken Moorhouse Silver badge

          Re: Cruft

          I thought it was a way of showing off one's doggy code.

  16. Anonymous Coward
    Anonymous Coward

    Refactoring is expensive

    It was Thoughtworks consultants working on our site a few years ago, who insisted that 'refactoring is cheap'. They would insist that since everything is test-covered, you could bash out the quickest, dirtiest, initial solution to each problem, and simply refactor it later if you needed to extend the implementation to support additional requirements.

    This was in response to my request that the initial implementation should be approached a certain way, given foreseeable requirements which we knew we would have to add later.

    Turns out I was right, as adding the later features bogged the team down in refactoring, which involved re-writing numerous tests in the process - thus compromising the quality of the original tests, when developers modified the test cases to make the re-written tests work. Additionally, developers began pushing back on making certain enhancements, stating that the way the early system had been written simply wouldn't support them.

    Unfortunately it seems that in the age of 'Agile' development, the idea of software 'Design' has largely been thrown away in the name of speed.

  17. disgruntled yank

    don't know

    ' "You could draw parallels with [Daniel] Kahneman's [book] Thinking, Fast and Slow... '

    You could, but do you really want to?

    """

    We computed the R-Index for studies cited in Chapter 4 of Kahneman’s book “Thinking Fast and Slow.” This chapter focuses on priming studies, starting with John Bargh’s study that led to Kahneman’s open email. The results are eye-opening and jaw-dropping. The chapter cites 12 articles and 11 of the 12 articles have an R-Index below 50. The combined analysis of 31 studies reported in the 12 articles shows 100% significant results with average (median) observed power of 57% and an inflation rate of 43%. The R-Index is 14. This result confirms Kahneman’s prediction that priming research is a train wreck and readers of his book “Thinking Fast and Slow” should not consider the presented studies as scientific evidence that subtle cues in their environment can have strong effects on their behavior outside their awareness.

    """

    https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wreck-how-priming-research-went-of-the-rails/

    1. John Smith 19 Gold badge

      Re: don't know

      Voted up for named post and citation.

  18. holmegm

    Hmm, maybe

    Hmm, maybe ... depending on what you mean by "good" and "quality".

    Yes, of course a lot of effort put up front into design pays off. But the devil is in the details, and especially in trying to systematize it.

    Just because the latest buzzword/framework/methodology/cult says that it is "more maintainable" doesn't automatically mean that it really is.

  19. Philip Stott

    The answer is easy as any fule knows

    I have been a contract software developer since 1998. I was entirely self taught, after finding my chosen field of interest, research chemistry, would be a life of drudgery unless I stayed in in school long enough to earn a Ph.D.

    So I turned to the other skill I had naturally, software development, by virtue of a family investment in a BBC model B microcomputer in the 80’s.

    Over the years I have mentored 6 six graduate trainees who wanted to switch to software development for a career (one of which, ironically, was a chemistry Ph.D that worked for Kodak and hated it).

    Of those six, I could tell within half a day which of them “just got it” (two of them) and which of them would always struggle no matter how hard they tried.

    There is a misunderstanding between the view “that anyone can learn to code” and “anyone can learn to code well”. It is an innate ability that one either has or has not.

    Long story short, is that companies that really care about code quality need to learn how to differentiate between these two types of developer.

  20. Anonymous Coward
    Anonymous Coward

    Software is Nirvana

    You make money shipping a buggy, defective product and then you charge more to fix it. And when you fix it, you introduce other defects so you can charge more.

    It's like going to a restaurant and find an insect in your soup and then the waiter will offer to remove it for an extra fee. With his dirty hands.

    I don't see any incentive at all to come up with good quality software.

    As Mel Brooks was saying in one of his movies, it's good to be king.

    1. John Smith 19 Gold badge
      Unhappy

      "I don't see any incentive at all to come up with good quality software."

      Unless there is an actual free market and you can dump your software for something else then you'd be correct.

      And any environment where the OS supplier is also an application supplier is not a free market.

    2. Ken Moorhouse Silver badge

      Re: I don't see any incentive at all to come up with good quality software.

      I like to sleep at night. I like to be able to answer the phone not dreading to hear the words "your program crashed". There is a terrific sense of satisfaction in writing something that works day in, day out for years on end without fail.

      So how do you make money in that scenario?

      Being invited back to increase functionality. If that functionality was thought about at the outset then it is easier to code in. Using Delphi means that, even if I've moved versions since then, the only problems likely to occur are in third-party components that have lousy upgrade policies (no names).

      1. Anonymous Coward
        Anonymous Coward

        Re: I don't see any incentive at all to come up with good quality software.

        Amazingly, I bumped into a former client I hadn't seen for years, and almost the first thing he asked me was whether I would consider adding some new functionality to a program I designed for him in about 1996.

        It has moved platforms several times and been updated, the last time five years ago.

  21. John Smith 19 Gold badge
    Unhappy

    Wow. 5 decades on and Glenford Myers is re-discovered.

    He called it "composite" rather than "top down" design as he felt that term was already over loaded.

    The least reliable parts of any system are the pieces of string connecting the black boxes, not the black boxes themselves.

    Myers work deals with exact;u that point. Especially poor is the system that passes complex data structures back that any function can alter any part of. IOW

    Every function needs to know its structure (and be re-compiled if it changes) and any function can corrupt any part of its contents.

    Which probably explains why Windows programming is so much "fun". :-(.

  22. John Smith 19 Gold badge
    Unhappy

    "this is not something you can fix in a couple of months by buying a fancy tool or something"

    Quite true. And that applies even if you bought the (usually) eyewateringly expensive training package to accompany it.

    OTOH

    "Software projects are always creating something novel."

    Oh really? That would make the idea of "software patterns" fairly pointless would it not?

    IRL a hell of a lot of this stuff has not only been done before but done many times before. Yet "developers" keep f**king up and behaving like code monkeys. I'll note (from "Programming Pearls") the first Binary Search algorithm was described about 1946 the first correct binary search was described about 1962. And that's for a "simple" algorithm which everyone here should be able to describe.

    If you define "novel" as "That which is the result of all the algorithms and data structures we've stitched together" that would be reasonable. Don't those algorithms and data structures already suggest some ideas for how to chop up the problem?

  23. Anonymous Coward
    Anonymous Coward

    I just came off an account...

    I just wrapped up a consulting gig at a company here in the states.

    When I was brought on board. This company was new to the Big Data space.

    They had a couple of projects already in the works and they had tight deadlines so it wasn't feasible to stop the work and start from scratch.

    One project was so bad, it would have been better if I had scrapped their code, and spent evenings rewriting it from scratch. But I wasn't in the role of developer and I don't work for free.

    While I focused on my role and level setting the environment, they started another highly visible project.

    Again this project is almost doomed to fail. I said almost because I was instrumental in getting a couple of guys who were capable and their work will be good, but can't speak to the work outside our scope.

    The real issue... the company hires cheap labor and these people can't do the work required. They mostly google for their solutions. In the end... what they produce is good enough to check the boxes but that's about it.

    This is the point of the article. This is why projects fail because companies don't hire competent staff who will write maintainable code.

  24. JoeySter

    I don't particularly respect Fowler as a whizz or an authority. To me he's just another programmer sharing observations and opinions. Though unlike me, he appears to have ultimately devoted more of his career to writing about programming. I'm not someone that particularly pays attention to his writings. I don't know what his code is like and I've only seen bits and pieces of his work. I can't say what I've seen is terrible but it is a single perspective and one person's set of opinions.

    I do however intimately know about the impact of his writings in scenarios relevant to what Fowler and those of his cohorts who follow the same philosophy propose here. I've had to inherit many a codebase "architected" by those who follow Fowler and have been given enough time or in this case rope to hang themselves. Whenever I see a monstrosity of a bloated OOP mess, a nightmare, there's a Fowler, a Gang of Four or similar book nearby. There are certain books that if you see them, they're a bad code smell, you can make a guess that the codebase is in a terrible state and you'll be right nine times out of ten. It'll be a specific kind of mess which might take many forms but inevitable comprises tangled layer upon layer of assorted abstractions. It's not that everything in these books is necessarily bad, it's a mix, though I'm not sure if the authors understand their audience, their audience's limitations and if it's even possible for any book to produce a positive effect.

    The problem is that people such as Fowler, whether intentional or not, create resources that lend exceptionally well to facilitating process driven blind programming that doesn't actually produce quality software. Fowler is a master of prescribing rituals that people who like to follow rituals can easily adopt.

    I can attest that in my experience writing code properly does result in greater productivity despite an initial lag as it has a much higher ceiling. The net result between the two can easily be an order of magnitude, sometimes exponential. Better code really does cost less. I've seen bad code now costing easily hundreds of thousands to millions over the course of a few years on programmers salaries alone. However, following Fowler has virtually no relevant bearing on whether code will be better or not. Nor will assessing whether code is "better" or not according to criteria that might be set out by Fowler either directly or indirectly be all that useful in ascertaining quality level. It's more likely to quickly depress software quality as people rush to implement things that aren't needed so to check all the boxes.

    Conversely people that follow Fowler and his crowd tend to make things much much worse when given more time. Fowler provides a lot of things people can waste time doing. More time means more layers of indirection and code for the sake of code. Virtually all of said additions incur a guaranteed cost and non of them provide guaranteed benefit. I'd like to hope that Fowler is aware of this and cautious in his approach, though his advice here has plenty of potential to be a plea to provide budget for this kind of waste which is counter productive.

    I might have a backlog like this...

    * Users->Feature: Allow users to have more than one X.

    * Government Agency->Compliance: Update data collection to include field X.

    * QA->Lifecycle: Make mission critical area X is easier to test.

    * Audit->Research: Investigate ways of detecting fake users.

    * Research->System: Upgrade database to next major version to gain feature X.

    * Benchmarks->Optimisation: Use precise indexes for use case X.

    * Business->Feature: Allow billing on an annual basis.

    While performing these, you might use a variety of different strategies and approaches as a side effect. Refactoring tasks might spawn from them.

    A back log inspired by Fowler and friends might look like this:

    * Fowler->Code Quality: Implement Specification Pattern.

    * Blog->Code Quality: Implement Command Bus.

    * CS 134->Code Quality: Implement Unit Testing.

    * Conference->Code Quality: Implement Event Sourcing.

    * Job Specs->Code Quality: Implement DDD.

    * Social Chat->Code Quality: Implement Visitor Pattern.

    * Best Practices->Code Quality: Implement REST fully.

    When a Foullower is performing these, they might implement some features as a side effect. Code Quality (tm) tasks might spawn from them.

    If you write a back log such as the one I originally wrote, don't expect it to solve the problem. All of the items from above will be aliased to those below, most likely respectively.

    The code produced by such a follower of Fowler will present you with a riddle, wrapped in an enigma, housed in a mystery, planted in a puzzle, kept in a secret and all tied up into a knot. It might as well actually be encoded which often ironically arises from people following blind processes for the promise of readability.

    I very commonly see code that only has to take a simple list from the database and return it to a client in a format such as JSON. If I implement this feature then I end up with very little code. Perhaps a dozen lines. If more than one object or method, almost certainly not exceeding half a dozen. We're probably talking around 50 lines, though do it a second time and that can drop to 10 lines. When it's code following rituals Fowler sets out, you can expect portions of it to be dispersed all through out the system. Dozens of hops, classes, methods, hundreds or even thousands of lines, etc. You'll probably find those things ill construed as ultimately effort will have been divided across such a system. It boggles the mind when people do things like convert a map to an object, send it through a maze then convert it back to a map again only to run a few if statements on a few fields and to rename or cast a few fields.

    You can't entirely blame Fowler for this stuff, people can take his stuff and do what they want with it or leave it but you also might think he'd be more aware of this set of problems arising from process driven and fad driven development. Not many people are going to like this but the truth not many people building software know anything about building software either and Fowler isn't particularly effective at improving that despite his efforts. Fowler exploring other shops is good, there's a lot of variety and things to see in software engineering, though I think he needs to see more environments, where he'll surely see that more time than really needed is catastrophic. Though it might not be obvious unless actually working on such a codebase and realising half of it can be tossed. Give away time and money don't expect it to be spent wisely by default. Some programmers will do amazingly but most will squander it. You'll end up with Albanian bunkers.

    I think there's also a cultural different. Many people including Fowler talk as if they've been spoiled in certain regards. It would be nice if that were the situation for all programming but the reality is, in the professional world that's by far not normal. Many programmers for not have financial security. Neither do their employers. It's not necessarily possible to pull an unlimited budget out of the air or support a rapid turnaround. The kind of thinking that might make sense for a cashcow such as IBM might not be appropriate for small to medium companies.

    I always strive to write high quality software upfront but also take the time needed to bring the software up to a minimum standard almost to the point of no matter what. What I don't do however is excess. Fowler provides plenty of ammunition for those prone to excess. There's always a competing axiom which is the more you do the more you have to do. Whatever you do ought offer the best value. The strongest drivers of cost effectiveness tend to be basic yet abstract principles such as YAGNI, DRY, consistency and good organisation and most of all being dedicated to purpose, not the the gospel of software engineering.

    Time is a secondary factor. Things have to be done to a certain level. If not then you lose the game, when you sacrifice the present for the future too much then you reach the point where you have no future. Two thirds of the projects I've inherited are cases where developers have done that then moved on at the point where it just becomes too much for them to handle, onto the next thing for a new start. Usually level two developers end up having to take over level two projects where level one programmers have not only done as much as they can but have driven themselves into a deadend. Individual skill factors in far more than other factors or checklist items.

    For me, in these circumstances, ironically code quality is absolutely crucial for a success. Elite teams taking "only an hour" to push a release? Asides from that not being qualified that's not by any means spectacular. This to me suggests a large discrepancy between people where it comes to experiences, expectations and standards It shouldn't be considered "elite" when a system facilitates development to production in under an hour. That's standard. If something takes an hour then there are usually special circumstances involved or it's just terribly constructed. I think to have a set of experiences where that's something to marvel only means he should get out even more to see what's happening in other shops.

    Personally, I don't do it by the book and I won't buy the book.

  25. Anonymous Coward
    Anonymous Coward

    Be careful who's doing what...

    Is this the same Thoughworks who outsourced the coding of a new retail system to India, went live with hundreds of unresolved issues and then when a customer tried to buy an item on day one the system wouldn't accept the purchase because stock showed zero? The company was turning over £200m and went the way of all flesh a few years later. Oops.

    Anon because I watched in horror as the car crash unfolded.

  26. hairydog

    Agile

    Unless you know what you are setting out to achieve, you are not likely to achieve it.

    Software design should start from a set of business requirements, gap analyses and functional specifications.

    The code should be based on technical specifications, with error handling throughout.

    The test scenarios should be based on business requirements and functional specifications. Not on technical specifications.

    Does this happen? Hmm. Sometimes. I saw it once, in the late 1990s.

    Since then, the "waterfall method" lost out to "agile" (aka make it up as you go along).

    How can you gwet it right first time when "right" is redefined as you code?

    1. Robert Forsyth

      Re: Agile

      Neither Waterfall nor Agile is one thing, it might be a case of using the right tool for the job.

      You build a machine in 3 months, it costs £1000.

      After a couple of months, you evolve/engineer its parts to save costs - a hand machined set of aluminium parts assembled into a component is replaced by something stamped out of steel sheet or injection moulded plastic. It now costs £700.

      After another couple of months, you redesign the sub frame, so that it can be assembled by robot. It now costs £500.

      After another couple of months, a key component has supply problem, you redesign for an alternative.

      Alternately, you build a machine in 8 months, it costs £500, but you cannot make any because a key component is not available. Anyway, you cannot sell any because a competitor has the market.

      Agile is a strategy to cope-with or manage change. Also it is finding problems early when they are cheaper to fix (LEAN); improve communication...

      1. Doctor Syntax Silver badge

        Re: Agile

        "Anyway, you cannot sell any because a competitor has the market."

        First to market does not always win. What happened to MySpace?

        Or you spend the 7 months building a nice adaptable design. The key component becomes unavailable. Because it's nicely adaptable you spend a couple of days adapting to a new one and crap all over the Agile guys while they're out of the market. You can do all sorts of stuff with hypotheticals.

      2. Anonymous Coward
        Anonymous Coward

        Re: Agile

        If a competitor has the market in 8 months, it's such a niche it probably wasn't worth the effort anyway.

        The idea that you must be first mover is quite wrong. For one thing, the second mover already has a waiting customer list of the people who are pissed off at the first to market's buggy product

        1. Anonymous Coward
          Anonymous Coward

          Re: Agile

          Or it simple moves so fast that you snooze, you lose. Look at how quickly Apple stormed the market with its iPhone and pretty much changed the whole cell phone game overnight. That's the thing with electronics markets. Their pace is breakneck, plus there usually aren't enough disgruntled first-movers to matter; whoever got there first is "good enough" enough to make them stop searching.

  27. Anonymous Coward
    Anonymous Coward

    I know loads about this subject and am right!

    Therefore I post AC.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like