Study finds 268% higher failure rates for Agile software projects A study has found that software projects adopting Agile practices are 268 percent more likely to fail than those that do not. Even though the research commissioned by consultancy Engprax could be seen as nothing more than a thinly veiled plug for the Impact Engineering methodology, it feeds into the suspicion that the Agile …
Stating the obvious….without requirements? Are you just going to code on your own, staring into space to decide what’s needed? Maybe it works for whatever hobbies you are working on. Your suggestion would be considered, shall we say “non-optimal” for an embedded microcontroller running the engine management of a passenger jet.
"for an embedded microcontroller running the engine management of a passenger jet"
Management is not QA. Never has been, never will be...having a management bonehead chirping in with dumb ideas every Friday afternoon because he had a brain fart would be considered non-optimal as well.
Management is not a key component of getting something built. Getting something built is a key component of getting something built.
If you get a bricklayer in to build a wall, you don't have a meeting after every fucking brick that is laid..you tell him the wall you want, where you want it and when it's built you kick it to make sure it's solid and built to spec. If it isn't you sack him and get a better brick layer.
Checking every brick will not get you a better wall or improve the brick layer because a shit brick layer is a shit brick layer...and you won't have a complete wall until every brick is laid...so even if you check every brick, you still haven't got a clue what the final product will be like.
Management overreach is just another form of bike shedding.
Yeah you have a senior bricklayer who has built lots of walls over many years whose focus is on the bricklaying itself. Not the overarching goals of the project. Not everyone has to be focused on the goals of the project as laid out by management. Its called checks and balances.
Managers are in something of a bind there.
If they fuck off and stay out of the hair of people who don't need managing it's hard for them to justify their inflated salary.
Or they don't fuck off and do actual work which is HARD (because they'd have to do BA/PM or hands on work) and RISKY (because they'll be a threat to other lazier managers)
I've seen so many good managers get laid off like that.
What the hell is wrong with you? Can you imagine the chaos if we started doing things properly, using common sense and some basic planning insted of simply following the current trendiest methodologies? My god, we'd have useless, overpaid project managers out on the streets despately holding signs with, "Will manage you IT project for food.". We'd start seeing projects completed on time and within budget, instead of being strung along twice the time and four times the budget, keeping swathes of over paid consultants on the payroll.
This!
Whole layers of management become redundant once an organization starts delivering software on time and with good quality. Watch the test and quality organizations panic when their months of planned testing is finished in a week or two because everything works.
Been there, done that, and it didn't require agile or scrum, though these would work fine and elements of agile naturally arise when doing things smartly.
Whoa, you never delivered on time with good quality? It's not possible, right? The vast majority of managers at all levels have never experienced this kind of result.
Plan for it to be hard work, but pretty boring, and maybe not as promotion-optimizing as full-on chaos. Project management tools that work com for real world examples of how to do this. I'm retired, but still like to point to our experience.
I've worked on projects with bad project managers, and it was chaos. I've worked on projects with no project management at all and it was just as bad.
I also worked on a project with no management that part way through hired a really good project manager and it made a huge difference.
I can only speak for myself, but I have a tendency to get drawn into the weeds if I'm actively working something and lose the big picture, I've worked with a lot of really good people who are the same. Having the grown up to make sure that we're all walking in the right direction and not about to wander into the road can pay absolute dividends.
But... there seems to be a lot of people with PM qualifications who have no idea what they're doing - or have ONLY project managed but have no concept of engineering principles or the like. Or even worse people skills than the engineering team.
I was bitter an cynical about the whole role - until I had a really good project manager.
Yes it was annoying and frustrating at first to be asked to shift focus, endlessly groom lists and all the rest - but once you then notice things start running like clockwork, no nasty surprises emerge, everything you've done has a bullet-proof defence... Well that feels nice. So next time around you're more trusting and the process becomes even easier.
And then they're laid off as the pointy-heads can't differentiate between good and bad managers - mainly due to the good ones not setting fires they might then ostentatiously put out.
I can only speak for myself
A mantra everyone should repeat before posting.
In my case, we switched from something like waterfall (though it was really an ad hoc collection of procedures, tools, and conventions used to implement release cycles that varied but averaged two or three years) to scrum in 2007, and it made the entire development process faster, more productive, and far more enjoyable. We didn't treat any of it as written in stone: teams were highly self-organizing and exact practices varied, with some top-down coordination so each team knew what other teams were up to, and we were all on the same release schedule.
Of course it could have gone badly instead. Small variations might have made it all a miserable failure. Flexibility and goodwill are critical, I believe.
But, no, Agile is not a silver bullet. There is no silver bullet. Agile isn't even a single thing; it's a set of broad, general suggestions that can be applied in lesser or greater amounts.
I haven't bothered looking at the report described in the article, as I doubt it's actually saying anything useful. (And thanks for that link back to the 2015 insane Erik Meijer piece; I'd forgotten about that bit of lunacy.) I will note, though, that we should expect a higher failure rate from Agile projects, because we expect them to fail sooner if they're going to fail, and be replaced by other projects. That might be better than spending years on a waterfall project which eventually fails. So the metric they're reporting is largely meaningless.
I've had just the opposite experience. We had scrum foisted upon us in 2007 as well. It resulted in endless useless meetings. The daily stand-up is just a daily status meeting for micro-management purposes. The sprint is an artificial deadline (with no basis on actual deadlines) one has to make every two or three or whatever length "sprints" you have. Sprint planning and story pointing is pointless (pun intended), because no one has a clue how hard it is going to be to get a task done (I always vote "3" or "5" randomly). Managers wring there hands over "burn-down charts" lamenting that there aren't enough hours left in sprint to get the work done based on the estimates. And while the powers that be are supposed to wait until the next sprint to inject emergency work into the team's schedule, that never actually happens. And of all they scrum variations I've used in the 17 years since were ALL equally dysfunctional for different reasons. This abomination has made a career I used to enjoy into a slog I can't wait to retire from.
An ale for the reference Babbage's Difference Engine. ;)
With a government contract to build the Difference Engine 1 he started in 1823 and abandoned two decades later but finally
"In 1991, the London Science Museum built a complete and working specimen of Babbage's Difference Engine No.2"
Apparently they haven't managed to work out how to build his Analytical Engine as the specification is unclear on certain critical points.
The novel The Difference Engine (1990) was rather amusing when read it around then but now seems almost a parody of the 21st century.
…but then of course recruiting, motivating, and retaining talented staff (at developer, manager, and product management level alike) is harder work and more expensive than sending a bunch of people to Scrum Master boot camp (and doesn’t look as shiny on an ambitious managers resume) so guess what happens.
I've been the industry over 30 years.
What we need is management and boards who don't buy into the "silver bullets" that the pundits keep pushing every year. They're nothing more than fads; the core of the job has changed little, though the tools have improved and sped up over that time. There are no "silver bullets" in this world, and that includes the incredibly overhyped, overpriced, and overspent LLM technology that is being pushed now.
In the end, I expect a raft of small business bankruptcies who blew their core funding on LLM garbage and got nothing to show for it, or a negative ROI.
That is exactly what will always happen if the PM is present at daily meetings, because they have power over the others present.
It may not start that way, but it is the inevitable endpoint.
If the PM is not present at the standups, then they become daily "worship" of the scrum master or similar role instead.
Regular meetings always become dominance games, because that's what social animals do. Standups are probably the most naked because they're every day, short, have no agenda and are in a situation where "loss of status" could be devastating.
Everyone feels pressure to either show off, or butter up the most senior person present.
If someone is stuck and needs help, do you really think they would say so "in public"?
I worked for a while alongside a project manager who became agile.
He organised 15 minute daily stand ups every morning. Then he found 15 minutes wasn't enough to find out exactly what everyone had done and tell them what to do next.
The stand ups grew to 90 minutes over the space of a few weeks. But they didn't have chairs, so it was still agile.
"Agile", in itself, says nothing about daily stand ups. Or sprints. Those things are not part of agile methodology.
They are a part of Scrum, which is a common methodology loosely based on agile principles.
Scrum was originally developed as a way of rescuing a waterfall project that had gone way off the rails, and salvaging at least some value from the wreck. That's the scenario of its early, impressive success stories. But it's very different from implementing a new project from the ground up.
But the agile methodology is now used as a bible. It's *the* way to run projects.
That's impossible, since there is no singular "agile methodology".
It's certainly conceivable that in your experience one approach has been used for all the projects you've been involved with, but it's not a universal truth.
Because current tech is polluted with accountants and project managers whose only role is to gatekeep engineers work to necessitate their existence. How many committees can you think of in your own organisation right now who conveniently keep their calendars full of meetings to appear present and useful in?
Puritanism is always a problem as by definition it is the only way and its adherants seak to purge themselves of any impuriies and regard all other ways as Satanic in nature.
This is not to sugest that there are no benifits from a particular approach. A hybrid approach may be effective posibly utilising a more conventional specification and project planning with threads of agile implementation - agile testing? Agile documentation? Agile UI development? etc as appropriate in the circumstances.
Dangerous talk…. and you’re absolutely right. I’m reminded again of the start of Jerry Weinberg’s Quality Software Management… reflecting how in 1956 “ “IBM was right. Thinking was essential.
But after a while I noticed that IBM and its customers often
honored thinking, but didn’t practice it. Especially in the
software side of the business, which always seemed to take last
place in the hearts of IBM executives.”
Excerpt From
Quality Software: Volume 1.1: How Software Is Built
Gerald M. Weinberg
https://books.apple.com/gb/book/quality-software-volume-1-1-how-software-is-built/id406436865
This material may be protected by copyright.”
On the topic of wetness, I was brought up on what became known as the waterfall process from before it had that name - it was based on not starting to spend on a project until you knew what you wanted. The preliminary phase(s) were the most trying and tiring; they gave the greatest hassle and a lot of project ideas never got off the ground. At one company I worked, the Capital Value Process (CVP) front-loaded multi-million $ projects such that the majority never got past the feasibility study; as a result, the company was very successful in the projects it did commit to.
Agile seems a way to dispense with the hard work up front, supposedly to reduce the early cost, but it comes as no surprise that there is a much higher failure rate - and, I suspect, a much higher failure cost. Actually, no need to "suspect"looking at government IT projects. Mind you, a high failure cost for the client doesn't stop profit elsewhere, so I'm not surprised Agile gets a lot of marketing presence!
Agree with the above. Moreover, if you took an average bunch of developers and ask them what they disliked about software development and then wrote up a methodology that didn't do any of those things, you'd pretty much have the Agile methodology. Developers don't like documenting things - so that's out. Working to specs - nope. Detailed estimation for the development? No, let's just focus on the next two weeks.
Agile is a methodology written by developers for developers. And unfortunately developers very often lose sight of the fact that the software is not the be-all-and-end-all, it's the use of the software, the purpose of the software, the benefit of the software that should be the focus - i.e. what the software does not how it was built that matters.
It's fine for certain types of projects. And absolutely wrong for others including, as a general broad-brush categorisation, most large business or Government type of applications. If you're a startup trying to launch your minimum viable product and then rapidly iterate, then fine, use Agile. If you're developing an ERP system or a finance/accounting/banking system, you'd better have a detailed spec, timescales, cost estimates, budgets and a clear idea of how to go about all of it.
The people who shout "Agile" from the rooftops have typically only experienced those projects that it (might be) good for and yet think that their experience encompasses all. Software development covers a huge spectrum amd in the same way that you wouldn't necessarily trust someone who can build a wall to build you a house, or trust someone who can build a house to build a skyscraper, what works or is appropriate in one place is not necessarily appropriate for another.
... coming from a software developer with 40+ years plus of bespoke development experience with a Computer Science degree before that. I've read, been taught, learned and used many different methodologies over those years, most of which promise to fix the failings of the previous one. Agile may have lasted more than some of the others, but I have no doubt that there will be another one along soon...
I think at least some things can be blamed on the devs. I have met few devs who like to write documentation. While I don't mind doing it too much, I'm always careful not to announce that because I don't want to write everyone else's documentation (so I do mind it at least somewhat), and I've been in a situation where I was asked to do so because I was the only one who documented my stuff without being asked ten times. I think the "working software over comprehensive documentation" line has been among the most damaging, and that sounds exactly like a thing a developer might have inserted.
If you're read The Mythical Man Month you can see what the Agile authors meant by "comprehensive documentation." And also, they final assertion in the Manifesto is that there is more value on the things on the LEFT but it doesn't mean there is no value on the things on the RIGHT. If good documentation is necessary for success in your project, why would you let a misunderstood "methodology" kill a requirement? (And Agile is principles, not a project management methodology.)
they final assertion in the Manifesto is that there is more value on the things on the LEFT but it doesn't mean there is no value on the things on the RIGHT.
Yes, I know. That has not stopped people from putting too little value on the things on the right, and we shouldn't be surprised because those principles were not elucidated. Never in their list of principles do they explain in detail how to do any of the things they suggest, and when people try to stick too closely to the principles, they end up overshooting the mark.
If good documentation is necessary for success in your project, why would you let a misunderstood "methodology" kill a requirement? (And Agile is principles, not a project management methodology.)
Because people didn't ask me. They decide what they're going to do, and they cite Agile as a reason if I question it. I am just one programmer on the team. I can't change it, not that what we need to change to is a similar thing with an easy name and list of rules. It's not that Agile is itself good or bad, because nobody does Agile the same way, but it makes some things very easy to do wrong.
> Never in their list of principles do they explain in detail how to do any of the things they suggest
Because Agile is a set of principles (what/why) not a methodology (how). One of its core principles is that those doing the work determine HOW they will do it most efficiently. Watch or read Martin Fowler's (one of the authors of the Agile Manifesto) view on the Agile Industrial Complex and why they purposefully avoided the issue of HOW.
If you're read The Mythical Man Month you'll find it said that the user manual is the first thing to be started and the last to be finished. It describes what the user will do with the product. If you're going to build something it's useful to know what the user expects it to do. The reason it's the last thing to be finished allows for the fact that whole project will evolve and that the final documentation will reflect what it became.
I've come across a fair bit of software which would have benefited from that approach - describe how the user will accomplish a task and build something that does that. Instead someone has worked out the various bits of functionality and presented them to the user in a heap. The heap, at first sight, looks logical - all the parts you'd expect to be there are there. But the user then has to work out what disconnected set of actions are needed to do what's needed because usability wasn't taken into account as it would have been if the objective had been to build something that followed a description of it being used.
I have met few devs who like to write documentation
In anything but the smallest of projects, I'd argue that the devs should not be writing the documentation. Writing good, clear, understandable by the users, documentation is a very different skill set to writing code - so should be done by people with that skill set.
Granted, there is quite a bit of overlap in that you need to understand a lot about the system if you are going to document it - but writing code and writing docs are different things.
Indeed. I have degrees in Computer Science, English, and Digital Rhetoric & Professional Writing. Those are different disciplines, and Technical Writing is yet another.
Developers need to supply the information that goes into software documentation, but very few of them are qualified to be writing the published documentation itself.
Now, there often is tension between the documentation team and Development in agile projects, because Docs may be faced with a moving target. But that's another sign that agile is being done poorly. Agile shouldn't be an excuse for user-facing features to be constantly changing, except for the sorts of projects where the vendor honestly doesn't care and the users don't get any documentation anyway, such as social-media systems. And it's quite possible to use an agile approach to produce the documentation; a fellow researcher and I gave a presentation on that at the ATTW conference some years ago. (Come to think of it, a chapter based on that talk was even published in a collection, though I can't recall the title now.)
When I work somewhere with technical writers to do that, I'll keep that in mind. Where I've worked, there aren't any, and if I ask why not, the same line might get trotted out again. If they can't be bothered to have me write a little of it, even though I don't have to explain it to anyone, why would they hire someone to do it? That would take an extra person and probably at least as much time from me.
Documentation is often neglected because time isn't factored in to write it, therefore it is out of scope. Also, if you have the kind of management that likes to change goalposts a lot, managing accurate documentation becomes incredibly difficult and insanely time consuming.
Poor documentation isn't always the result of devs not wanting to do it (although that is a common factor), it's sometimes due to it being an afterthought.
I've worked on projects where basic documentation was written before a product even existed...which is an awesome way to work because it adds a level of clarity you can't get from a spec or a Miro board.
This is a method that works particularly well for designing APIs.
I once worked for a project manager whose idea of documentation was a two-page (max) description of the software being used. IMHO, that was barely enough to list all the software components/languages being used and, maybe, a terse description of what it was used for. But two pages was his limit.
> Agile is a methodology written by developers for developers.
Also this does not chime with my experience (developer > 45 years) whatsoever. Replace "developers" with "BAs", though. and I think you might have it.
The issue I've found is it isn't the actual job of the developers to put the requirements together and to create documentation. The BAs need to define the business problems and put these together in a document as a set of functional and non functional requirements. The IT architects then work out the solution at the high level and create a set of architectural design documents.
Only then should the developers get involved. Of course there always must be feedback from the dev team back through to the architects and BAs along the lines of "what are you smoking" to getting things to work in the physical reality. You also have a QA team that are responsible for testing. I've found that it is never a good idea for a dev to do the testing as they will test the software in the way they think it should work. What you need is an extremely deviant QA that just loves to see how they can destroy a piece of software.
And of course the documentation is under the control of the technical writers.
Under Agile I've found the BAs put something half arsed together the throw it over the fence to the developers and say build.
In my experience Agile is useful for projects where you have close frequent contact with an engaged customer, and most of the functionality is fairly shallow i.e. each button or interface does something fairly simple/small. What it doesn't work for is large complex functionality where the customer has no interest in the deep workings of what's going on, and just wants a simple interface which hides it all.
For my first Agile project I was the (sole) developer writing a database upgrade framework that was going to take about 2 months to get the core engine working. At the initial meeting (of eleven people) for the first sprint the product owner wanted me to generate user stories, the documenters wanted initial documentation from me, the testers wanted test plans, and the project manager said he wanted a prototype GUI with a button saying "Start upgrade" on it. He said he didn't care what the engine did when it was pressed.
I told them that I wouldn't have anything to demo/document/test for at least a month, which apparently made me "not a team player"!
You are correct. Agile works when you have good communication with a focused user.
However most projects have several “involved parties” most of whom think you are wasting their time.
A simple on line ordering system involves inventory, ordering , warehousing, transport, HR, legal, accounting, tax etc etc. and the visionary genius who thinks they can be the next Amazon.
The most important BA skill is diplomacy. Trying to get a warehouse manager tell you how he is going to organise 20 box fillers he doesn’t employ yet, in a space that hasn’t been allocated yet for a system nobody asked him about.
That tracks with my experience. I haven't been in very many Agile projects, but of the three, two failed.
I'm sure that proponents of Agile could pop in and say that it only fails when badly executed (e.g. when you skimp on unit tests), to which I would observe that, by that line of reasoning, abstinence would be a good contraceptive. If a few people misuse a methodology, that's a problem with a few people; if a lot of people misuse a methodology, that's a problem with the methodology.
This is just my gut feeling, but I can't help forming the idea that Agile is a well-crafted justification over the obsession with "now now now" and "planning for more than the next thirty seconds is boring" that is so typical of our age.
"This is just my gut feeling, but I can't help forming the idea that Agile is a well-crafted justification over the obsession with "now now now" and "planning for more than the next thirty seconds is boring" that is so typical of our age."
See to me, Agile is the counter answer to the phrase "Poor planning on your part doesn't constitute an emergency for me".
Agile -
1. Do stuff really fast & half test some of it, some of the functionality even works and release
2. Do more new stuff really fast and if something from step 1 really needs fixing, look at fixing it, half test and release
3. Some bits broken in step 1 now work, some bits that used to work in step one are now broken, some stuff in step 2 works but PMs and Devs are already working on the next pile of shite to release so what is already happened is deep history.
Usually the projects produce just enough usable stuff that more or less works as intended for it to be considered a success
Rinse and repeat.......
Your comment shows you don't really know what Agile is, and that people have stuck an "agile" label on your projects without knowing what it was either. Agile is a set of principles, not a "rinse and repeat" process. It's not even a project management methodology! One of its principles is working software. If it doesn't work because you did it really fast and only half-tested it, then you are not Agile; you are an undisciplined and unprofessional developer.
One of its principles is working software
Well, Doh!!! You actually got that right, the issue with Agile is that the software might work. BUT, it doesn't actually meet the completely unknown requirements because these requirements didn't exist when it was created!
If you know your destination then you have a chance of getting there. The problem with Agile is you know you have a journey but you have know idea as to the end point. And half the time you start in the wrong direction.
In the software world you create a whole lot of code that you end up throwing out as in one case for me the database subsystem chosen was not fit for purpose, good 9 months wasted.
This post has been deleted by its author
Abstinence is 100% effective, a better success rate than any other pregnancy prevention. But to achieve that success, it is necessary to actually follow the rules for it, which is why it is so unpopular as a way to prevent pregnancy.
Unlike Agile, where there are few rules and no thinking ahead, and thus lots of failures.
I think it's unpopular because when someone says they want something that will prevent pregnancy, there is generally an implied additional requirement of "while having sex".
This makes the whole thing a pretty good example of why it's good to define all requirements clearly and explicitly up front, and how a hidden requirement that only appears later might end up invalidating the entire basic design in such a way that no iterative improvement is going to fix it.
Customer wants x, by y for z amount.
Massed developers wander off for months for a group code wank.
They then present their completed shit to the customer and then blag it when asked questions.
When the created thing doesn't work, the customer gets the blame.
IMHO it doesn't matter if it's agile, waterfall or whatever, the customer has to be central to the whole thing and not an afterthought.
> the customer has to be central to the whole thing and not an afterthought.
That is *supposed* to be of the basic tenets of Agile, with constant contact with your customer.
Of course, if you ask for a customer contact to attend your Sprint reviews on a weekly (or even monthly, for the arthritic agile) basis, are you going to get their knowledgeable person, who knows their business inside and out, or is she too busy keeping them functioning? So you get the young chap who was hired because "knows about this Agile stuff" but bugger all about the workings of the company that he only joined last month?
Always assuming you actually have *a* customer for this project, and you aren't doing anything silly, like, ooooh, making a system to fit a general market niche, with a dozen paying customers already lined up to get copies configured for their individual situations; 'cos who is a *good* "customer rep" for your Agile Meetings now? Do you rotate them, one a week? Or do you have one of our own guys, who has created a single set of requirements that sort of mixes them all together - but you don't work to those 'cos you are Agilely building it piecemeal and hoping to converge on customers' reality at the end? Even better, do you have an external third-party Agile Expert playing that role, with the benefit that he is equally ignorant of both the customers and your dev team and your previous products whose components may be reusable?
No customer wants "sprint reviews".
What they want is "we'll have x done by Friday, and we'll demo it to you; y will be done a week later; z will be the end of next month and that's the job done."
Clients do not understand agile, nor should they need to. If you were building a house would you expect to have a standup meeting at 6am every day with your builders to review what's happening that day? No, you'd leave them to it for the most part and look for deliverables.
"If you were building a house would you expect to have a standup meeting at 6am every day with your builders to review what's happening that day?"
I don't know if this would be such a bad thing... I think it would depend on how this was executed. A short review of what was done yesterday and what's going to be done today might discover (and prevent) potential minor problems before they turn into a major problem. Whether it's a house being built or software being written. At a minimum it lets each person vent a little (if they had problems the previous day) and gives the supervisor a little insight as to how the overall job is going, as long as they aren't judgemental about it. Would this be a bad thing to know? I hope not.
Yes, a brief daily review is a good thing - but that's the supervisor's job, not the client. The client is paying for a building with specific details, built to code, and should not be expected to have to babysit the contractors' meetings. (Otherwise why should the client not hire the workers themselves?)
That's when the client has been clear about what they want before construction started. If they haven't, then you have a different situation that more closely matches the typical software development process. Agile and the developers and managers who recommend it can take some of the blame for that, but the customer gets some of it too.
Customers often say "I want X for Y amount in time period Z", but X is often very brief. X might just be "a database to track [something]". The devs have to figure out how to turn a generic request into something that actually does what the customer wants. The analogy to construction is more if I just said "I want a house. Four bedrooms. Get started.", but after they had built something, reacting with annoyance that they hadn't built in the style I had imagined.
The customer can specify all these details up front, but some don't try and some try and fail to do so properly. That's why I prefer to have them in the loop, testing intermediate versions, so that I can build what they want instead of what they said they wanted.
@doublelayer: There is the flaw, specially in large corporates including governments, that the user area knows what it wants but manglement get involved, especially higherups looking for a resume aimed triumph , who then add crap. Loss of focus, followed by changing requirements from PHBs usually, followed by user level failure.
I don't think that's fair either. Users know what they want, but even if you remove the managers and talk directly user to programmer, which I have done, you don't get the information without lots of effort from both parties. They understand in broad terms what they want, but there are two important gaps:
1. They only understand the surface layer of what they want, not all the components needed to build it. By components, I don't mean the technical details, but parts of the plan. They understand that the following pieces of data need to be collected, and stop there. They don't think about what the computer should do with the data once it has been collected. The form to collect it is what they can see, whereas the backend automation is not, so it often slips their mind. You have to remind them and step through the process manually to figure out all the things that should happen to the data once you have it.
2. They understand at least that part of the need, but they are not good at expressing it. Even for the parts they have already planned out, they tend to omit crucial details and assume that, because they know it, the programmer knows it. You have to break this down and ask lots of little questions, or you have to make assumptions to fill in the gaps. In my experience, the best relationships come when the programmers are capable of making those assumptions and it works well, but the worst teams are when the programmers aren't capable of doing it and do it anyway, so you have to be really careful at that line.
There are two approaches that tend to work with this. You can plan out every detail of the program through weeks to months of painstaking, boring meetings between the devs or someone dev-adjacent and various layers of the customer. The customer doesn't really want to do this because they don't understand why so much detail is necessary. It's not the fastest or most efficient, and there will inevitably be some gaps even at the end of this. It does limit the number of gaps at the end.
The other is a more agile method, and one I prefer if you can get it. You have some sessions with the customer to make part of a plan, build that part, and continue to iterate. If you have a customer who works well with this, then you can build things quickly and pivot fast if their requirements are different than you imagined. However, it does require that people who know what they're doing continually test it. The developers or QA if you have one is testing to make sure it doesn't break, but the customer has to test that it does what they expected it to do and that you know if it doesn't. It is easy to just assume that the customer is testing it and take no news as good news, but not the best method.
You can have them as long as you take the management with them. The users have a limited understanding of what they want, which is not necessarily the same as what would be of most benefit to them, but making the time to properly understand what they want and why they want it before I start writing tends not to get prioritized. Fortunately, I'm no longer in the position where I was simply given tasks without anyone explaining who wanted it or why, missing all the important details and I couldn't figure out who to ask for them.
"You have some sessions with the customer to make part of a plan, build that part, and continue to iterate."
A third alternative. You build some sort of prototype - possibly even a sketch of the UI with a note about what does what. You present that to the user. Once they see something that may vaguely resemble what they were looking for they're better placed to think it through and clarify. But don't put effort into building part of it properly because even if it shouldn't be part of the finished product you're stuck with the sunk costs.
The user area might know what it wants, but can't effectively communicate it to management (or anyone), and you as the dev hear it third hand off a manager that neither understands what the original users wanted nor how to communicate it.
If you communicate directly with the users, you might be able to work something out. If you work with the users through the manager, all you get is the sound of a chimp chewing his tongue.
The analogy to construction is more if I just said "I want a house. Four bedrooms. Get started.",
Someone who wants a house either goes to an estate agent to buy one that already exists or they go to an architect* to get a design put together. Only then do the builders/devs get started.
* Occasionally someone will want to be their own architect
Exactly, and that should be what they do with software as well. Either you need software that has already been written, and if you can't find one then you can ask someone to help you find it. If you can't, you should go to someone who can take your requirements and help design what you need. For instance, I know a guy who needed to have a custom-designed house because he uses a wheelchair. Sure, you can retrofit an existing house and make that work, or you could move to a place where houses are already well-designed for it, but if you don't want to deal with the difficulties each of those brings and you can afford it, then designing one from scratch can work. It would not have worked if he failed to mention that until the first designs were drawn up and someone was building it.
Unfortunately, that happens all too often when people are describing software. They assume that you know that there is a requirement that they didn't state, or they think they stated it and probably did but not in sufficient detail that you understood what it meant. It doesn't even have to be something as big as my analogy. This is why you either have to figure that out at the start or stick with the customer through the development, constantly getting their approval or feedback on things. A lot of people who claim to do Agile aren't doing that latter version, which Agile recommends, and therefore they don't get what the customer wanted. There are a number of reasons why they don't, and one of the main ones is that the customer doesn't want to. However, they need to recognize that they either need to make the customer do it anyway or they need a process that can work without it, and skipping it but continuing to use the process that requires it leads to substandard results.
"If you were building a house would you expect to have a standup meeting at 6am every day with your builders to review what's happening that day?"
No, you'd have building regulations to follow by law and an architect who'd done the planning and design before any trench was dug or any brick laid. The problem with pseudo-agile is that neither of these requirements are met. There are no binding standards to follow, and nobody produces a complete plan or design up front.
A supervisor or inspector might have a meeting like that, they might not. But with the client? Someone the limit of whose knowledge is that they know one end of a drill from the other but that's more or less it?
A lot of agile has, to my thinking, always suffered from the same conceit as economists do: assuming everyone is a perfectly informed actor with omniscient knowledge. When in fact this couldn't be further from the truth.
Have you ever built a house or had major renovations done? Ideally you would do exactly that (6am stand ups daily) but it's not always possible. You as the owner have to be on top of everything, or have a good and trusted project manager to delegate to. You also need highly detailed plans speced right down to colour of the door hinges. Once built you can't just spin up a new version if a house to fix structural problems.
"If you were building a house would you expect to have a standup meeting at 6am every day with your builders to review what's happening that day?"
I have actually used building a house as an example how how Agile fails.
We are going to build a house, ok first lay a slab of concrete. Next some walls, oops we forgot the plumbing for the bathroom oh and the kitchen is in the wrong corner of the slab. Take the walls down dig up the slab and start again.
Seriously, I have seen this several times in an Agile project, Start of half cocked before requirements have been correctly defined. Then just wing it.
Agile is disposing of the part where the analyst/project manager talks to the customer and writes down the requirements. Instead they expect the programmers to talk to the customer as they go along while making things up. This is quite convenient if you are an incompetent project manager.
They do have to be central. After all, they will be paying for and using the system.
But the customer needs to be controlled somewhat, otherwise different people will have different ideas of how the system should work, and in trying to cater for everyone, the developers will build a system that doesn't totally satisfy anyone's needs..
What if the customer doesn't understand what they're asking for?
Customer: Yeah, so all I want is for people that visit my website to be able to connect their smartwatch (whatever it is) and upload the data to us so we can analyse it. It's that simple.
Dev: Ok, well that's going to be a bit complicated.
Customer: They're all bluetooth though, so how hard can it be? Can't you just get the data from Bluetooth somehow?
Don't forget that to a lot of non-technical folks, completely different hardware etc can appear to be identical because they don't know how it works...so a "simple" idea they've had is actually a nightmare.
Maybe the sort of people who *really* get into Agile are simply the sort of people who couldn't successfully complete a project in the first place?
Where "successfully" includes "it does still work a year later, after we've left and you have had to actually use it yourself, including following the documentation (!) for the six-monthly reconciliation run that we never actually tested with 100% real data 'cos we never had a stable codebase for that length of time - oh, and when we couldn't remember the name of the config options we obviously just looked at the code, that is what self-documenting means".
> Maybe the sort of people who *really* get into Agile are simply the sort of people who couldn't successfully complete a project in the first place?
Or more likely (and this is my usual experience) that people who think they are doing agile, are not really doing agile. They do the easy bits (eg. avoid lots of up front documentation) but not the hard parts (eg. effective feedback process on how things are done; the prioritised feature list is actively maintained with the customer deciding when enough is done in place of a fixed budget).
Ie. almost every agile project is using the easy parts of agile to avoid the hard parts of waterfall, and thus is the worst case of both.
"customer deciding when enough is done in place of a fixed budget"
One software project I was on had the customer deciding when enough was done _because_of their fixed budget. It was awful. I was on the project as a kind of reviewer, technical friend to the customer and technical reviewer to the contractor. I spent most of my time telling the contractor that the customer's requirements were rubbish and how did they ever expect to deliver against them. Contractors response was typically "we know but the customer keeps changing their mind so we keep starting again". The other times I was telling the customer that despite them telling the contractor to use Agile (I think the customer had been on a course) they weren't doing Agile. Customer's response was "I didn't mean Agile,I meant agile, fail fast". I really wanted it to succeed but when the customer declined to continue paying for our services (because we kept telling the customer what they needed to hear not what they wanted), I didn't cry too much.
are simply the sort of people who couldn't successfully complete a project in the first place?
> Or more likely (and this is my usual experience) that people who think they are doing agile, are not really doing agile.
Agreed.
Looking back, I should probably have put in a couple of quote marks and some capitals to express it better (cf my other comments here):
>> Maybe the sort of people who *really* "Get Into Agile"
"Maybe the sort of people who *really* get into Agile are simply the sort of people who couldn't successfully complete a project in the first place?"
Oddly there are people who build their whole career on never finishing a project: They get it started, sure, and make the project seem like it's succeeding, but they move on before it's close to delivery and people find out it's not fit for purpose. They then take the key people from the project to get started on the next project meaning the first project falls apart and fails: Not due to those who moved on, but because those who took over weren't able to keep the project on track (honest!). Never the fault of the initial team, and no way to hold them accountable for the terrible job they'd done, whose faults were just tucked away and not spoken of because - wasn't going to be that team's problem when it came to delivering the goods.
The NHS were lousy with people like that. It's why many projects failed, wasting time and money on what often was just a vanity project, or alternatively something to keep mates in jobs.
A lot of the people don't get to choose their paradigm in any case. As a developer, my management decides what things will be, and they don't ask for my opinion. Also, it's not as simple as selecting one from a menu. If choosing something else with an easy name was an option, then maybe we developers could get together and demand that "We refuse to use Agile. We want NewCoding." That doesn't work because there's lots of pieces of Agile and no team actually uses a standard version of them. They use whatever they already want to do and call it Agile.
That might be because there is no concretely defined Agile process (very Agile, that). You can mix and match processes from Scrum, Kanban, Lean, XP, and whatever else comes along next.
There are no rules, but tenets or guiding principles or whatever it's been sold as in the 3-day Agile course that everyone's been made to go on so that the company can now say it's Agile.
In the end the customer has a limited budget and has to be able to explain to developers what they want so the developers can build it. Both are not easy problems to overcome but both can handwaved away with Agile, at least until the project fails.
Whenever the agile/waterfall argument surfaces, most often the definitions of both are stretched to the extreme -- one single all-embracing Niagara versus a bunch of devs coding while trying to work out what the application should do. Neither is very realistic, or indeed fair to either methodology.
The ideal approach to waterfall (particularly as we now pretty much assume object orientation) is to first design the entire project as a set of functional module descriptors then pass these modules to individual design processes (and where necessary, do the same again at the next level of detail) -- resulting in a cascade of small waterfalls rather than a single huge deluge.
Agile at best is an interactive approach to the definition of functionality refined by feedback from prospective users. It's most applicable therefore to the development of user-interactive components of a system such as the UI and information processing, and less appropriate to hidden functionality (e.g. security and performance) about which users will not be able to provide significant feedback..
So the two methodologies are not mutually antagonistic, but complementary.
The big problems we face today are [1] an assumption that agile is universally the 'best' approach and 'waterfall' is 'legacy', [2] a perversion of the original intent of agile, manifested as leaping to the keyboard and coding without clarifying the intended functionality first, and [3] assuming that documentation and version control are unnecessary. The result is just chaotic uncontrolled coding,. The manifesto did not advocate this, so it's unfair to call it agile. It's equally unfair to dismiss 'waterfall' based on assumptions about the most inefficient way to implement it.
While I agree with all of that, I think it's also necessary to consider the scale of the project and its intended lifetime.
Agile is only intended to work with small teams, so its scope is limited in any case. Its "locker room" concept also originally envisaged the team being physically adjacent and separated from other teams and in that respect very reliant on spontaneous communication within the team to compensate for the lack of top-down direction. Unfortunately, many software developers are not spontaneous communicators. There's almost always some additional management infrastructure that's going to be needed. And there is still generally a view in upper management that investment should be quantifiable and have an end date: the idea of successive and indefinite refinement is a difficult one to sell.
I'm very much a believer in adapting the process to suit the strengths of the people who will be doing the work. There are people who work well on their own, people who work well in teams, people who are good at prototyping and people who are good at solid and reliable coding. And you need a different approach if you're outsourcing some proportion of the work. No-one is going to offer you a certificate in pragmatism, however, so it's not a recipe for getting a project management job.
The downfall of "methodologies" is that they assume that everyone plays the game. I was talking a while back to an Agile advocate who admitted that in most cases they struggled to get the product owners (the people responsible for defining the requirements and acceptance of the resulting system) to adequately perform their role - they tended to regard IT as someone else's problem. Unfortunately, as the notional commissioners of the work, they are the effective customer giving the software developers little leverage over them. If you insist on managing your projects with a dependency on key stakeholders in the full knowledge that some of those stakeholders will not adequately participate in the process, you're doomed to failure. If you don't have an organisation that can insist on cooperation, then you have to adapt your approach accordingly.
While I agree with you, we both must admit that either methodology has lost the primary aspect in both cases : talking with the customer.
I was doing Agile before it had a name. I'm a Notes developer (silence at the rear). The first thing I learned when becoming a consultant was : the customer doesn't really know what he actually wants. My job, as I learned it, was not only to listen to the hierarchy and document its needs, but get the view from the groundpounders (the ones who will actually use the product) and learn what their needs are and how to best fullfill them while placating the management.
I don't care what methodology you call that. What I see is that I have now spent 25 years of a career where I can say that my successes have vastly overweighed my failures. I have helped people do their jobs better and more easily, and I am rather proud of that.
Definitely this. It's extraordinary the number of potential customers who don't seem to understand how their business actually works - or don't want to admit to the messy details. Or do understand the minutiae of current processes and insist on them being replicated exactly, even where they're causing frustration at the coal face. Negotiating your way through this can be a diplomatic feat that dwarfs the technical implementation.
While I agree with all of you, I'm going to put up another long winded post that refines the niggles that blame the customer/management/developers for being the sole/joint/collective cause for the overrun/overbudget/failure/global-warming of the project/NHS/welfare-state/colour-of-the-teletubbies!
TLDR! Just assume I told a good joke parodying all of the previous posts!
----------> Mine's the one that gets a new set of patches every 4-8 weeks!
Edit: Patched to fix speeling mistooks!
Agile is ... intended to work with small teams that have all the disciplines & resources they need
Need hardware? The server team want sizings & sufficient lead times. Unless they have a lot of spare capacity just sitting around (Or pay a premium to use Cloud resources because they didn't know what you would need).
On the upside, every time I'm told "here's what we want*, and you're now blocking our project", I get to quote the 6 P's
[*] i.e. what we think we want today; It'll probably change tomorrow.
Agile is only intended to work with small teams, so its scope is limited in any case.
That's not my experience. We've been using it for 17 years on a large, complex set of products with more than 30 teams, with wide geographic distribution. The teams are self-organizing, but they're all some variation of agile. There are product managers who do the market research and request major features (but many of the features come from within the development teams), and project managers who coordinate among teams for release tracking and so forth, and there's a single tool for managing work items. Products come out on schedule (annual major releases and monthly updates).
It's really not that hard.
The Manifesto is fine. The problem is how the methodology is sometimes applied.
You don't start a project without requirements, period. That is as true in agile development (note the lower case "a", because using it as a Noun is where the trouble started) as it is in every other methodology. Agile development is about incrementally going towards a goal. If you don't know where that goal is, you are not going anywhere, because what yard stick do you measure your progress by?
> The Manifesto is fine. The problem is how the methodology is sometimes applied.
Not helped by companies whose management read *about* the Manifesto and then start the project with a big meeting all about "The Yoyodyne Approach To Agile" which ends being neither one thing nor the other, with roles[1] such as "software architect" turned into positions on the org chart[2].
Not to mention (but I shall) redefining "software architect" from keeping track of all the little bits[3] to "The Guy With The Big Vision"[4]
[1] which can be filled by different people at various times
[2] which are used to set Titles On Business Cards and salaries and "progression in the company"
[3] everyone is using the same library to do the same job, oh, and version n, not (n+1), because n runs the same on all the different bits of kit involved, here are the tests to show that
[4] do not bother me with piffling details, of course you can use XML DOM on a memory-constrained embedded system, SAX has got a silly name!
Proper communism works fine, in small groups. Too large a group, though, and leadership is necessary, and the leaders (in the long run) inevitably acquire more power and resources than the workers - making it socialism.
Only in the short term at best.
Long term, if you're lucky then the group will increase in number - then become unstable and turn into a socialist dictatorship.
If you're unlucky, it fails in a different way.
> The same has been said of Communism, of course.
And maybe it is. Maybe it isn't.
We don't really know, because real Communism has never actually been implemented as a socioeconomic system at significant scale, anywhere in the world. A lot of countries pretended to do so, only to end up as autocracies or oligarchies.
Personally, I think Communism, at least as postulated by Marx&Engels doesn't work, because of how human societies tend to self organize in a way that makes the idea impossible to really work.
But, again, we don't really know.
"The Manifesto is fine. The problem is how the methodology is sometimes applied."
I'm not sure I can agree. The application is most definitely responsible for some of it, but the manifesto provides their excuses and doesn't even try to direct them away from those misapplications.
In larger forms, the manifesto says a lot of obvious things. "Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done." That's great, but it doesn't actually tell you how to do it, so it doesn't tell companies that they're doing it wrong. Where the manifesto does make a recommendation that can be applied, it's things like "Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage." Of course, a project shouldn't ossify and refuse to make minor changes at the end because it's the end, but this statement is more extreme and is frequently interpreted as "start over whenever the customer asks you to even if it means wasting months of work". Then they act surprised that it's months later than the customer wanted it. This is more likely to happen because they read "we have come to value: [...] Customer collaboration over contract negotiation[,] Responding to change over following a plan" and so they don't bother getting the design right in the first place. The customer's happy, because they don't like doing that design process. They see that there are new builds every few weeks and that seems great because "yay, there's progress", but they don't test them. Developers are happy because the customers are getting what they asked for and surely they're good with it, after all that design's been in the test versions for a couple months now and they've not complained once. Then the project extends six months over asking, everyone gets unhappy, and the contract determines who has to pay for this and someone gets very unhappy.
The manifesto, without being explicit about these things, is at least partially responsible for this. If the people who wrote it knew about these problems, then they failed to describe or even mention them in their writing. If they didn't, then they probably weren't great at designing a new paradigm. The companies who applied it are responsible for each of their failures, but I don't think it is fair to say that they could have applied the manifesto properly and gotten around their problems.
> but it doesn't actually tell you how to do it,
True, because that's not the manifestos goal. It points out an, at the time, alternative and new approach. Its a guideline and Idea, not a Step-By-Step cookbook on how people should do their job.
Agile software development + Rigid rules how to do it, is kind of contradictory. And, I believe, it is this contradiction which led to a lot of people viewing agile development as bad...because what they perceive as bad isn't agility, it's just another set of rigid structure imposed on them, calling itself "Agile" (notice how it suddenly became a Noun). Suddenly they no longer get the "environment and support they need", and their supervisors no longer "trust them to get the job done".
Instead they suddenly find themselves as little squares on a giant flow chart some consultant presented to middle managemet for ungodly consultancy fees. Suddenly they drown in KPIs, and standups, and sitdowns, and have 3 meetings per day. Suddenly they need approval from 7 different people before a small CSS change is allowed onto production. Suddenly writing 17 new test cases because a param changed from uchar to uint8 is now somehow Agile (again, it's suddenly a Noun).
That is not agile software development however. That is a mixture of corporate bureaucracy + buzzwords, labeled with a noun that makes no sense (I'd like to have 12 pounds of Agile please).
"True, because that's not the manifestos goal. It points out an, at the time, alternative and new approach. Its a guideline and Idea, not a Step-By-Step cookbook on how people should do their job."
You're trying to let them off the hook again. It's not a guideline. It's at its most benign a statement of the obvious. It's as if I made a new code of moral ethics which reads in its entirety "be kind to people whenever possible unless absolutely necessary". You can't argue that my code is wrong, even if you think it probably is, because I have not bothered to define "kind", "necessary", or even "absolutely".
At its less benign it continues to be hopelessly vague but does make some recommendations. Recommendations opposed to plans, documentation, and formal processes in general. Without expressing any limit, reason, or details, they should not be surprised when those recommendations are followed too strongly. I'm not even sure if they understand that's a problem.
When I say rules, I don't mean about the structure of the thing. Agile doesn't have to talk about when and how you hold meetings or what the organizational chart needs to look like. That is not in the scope. But if they say that they want people to welcome changes late in the process, they should absolutely specify how they think that should be balanced against the fact that, if you change everything late in development, you will not be done by the original deadline. Should you limit the changes you accept late? Should you be open to moving deadlines? Should you maybe try to prevent there being so many changes by doing something earlier in the process? If so, what should you do and when should you do it? They should have answered these questions if they wanted to be understood. Since they did not, misunderstandings are partially their responsibility.
> You're trying to let them off the hook again.
There is no hook. The manifesto is not required to give precise instructions, period. It's a couple of good ideas that some people wrote down. That's it.
> It's at its most benign a statement of the obvious.
Is it though? Because, at the time it was written, it sure wasn't obvious that this could be a way to build software. Most people at the time were still doing watefall.
> Recommendations opposed to plans, documentation, and formal processes in general.
No, they are not. The manifesto prohibits neither plans, nor documentation. Nor does it prevent it's implementation as a formal framework, otherwise things like Scrum wouldn't exist.
What the manifesto does state, is that none of these things should stand in the way of the only KPI that actually counts, which is to deliver functional software to the user/customer, and deliver it as soon as possible and as well made as possible. When I could give the customer a feature next week, but cannot, because some bureaucratic process requires me to write 30 tests and 7 pages of documentation for every minor change and have it evaluated and signed off by 4 people, 2 of which are on holiday and one needs the change as a printout on his desk no later than 11am, then the modus operandi is not working out as well as it could.
I am exaggerating here of course, but this is still not far from the situation that many teams experienced back then. This is the kind of problem that the agile manifesto first recognized, and that's exactly what it's recommendations try to prevent: By dialing back some restrictions, and free developers from bureaucratic obstacles, to make delivery easier and developers happier, and deal with whatever minor problems this causes later.
> Should you limit the changes you accept late? Should you be open to moving deadlines? Should you maybe try to prevent there being so many changes by doing something earlier in the process? If so, what should you do and when should you do it?
The manifesto DOES answer these questions:
"Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done."
"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."
Because the answer to all of the above is: "It depends". There are projects where late changes can be brought in, no problem. With other projects, some changes can be brought in, others have to wait until a later version. And then there are projects where the deadline is so critical, the team absolutely cannot change anything after a week. It depends on the individual project. The team knows its own project. Therefore, the team decides such questions.
"There is no hook. The manifesto is not required to give precise instructions, period. It's a couple of good ideas that some people wrote down. That's it."
The manifesto is required to explain itself if it wants to be understood. It does not. It is required to give enough detail in its ideas so we know they're good. It did not. Your argument is as easy to defend as saying that it's a collection of bad ideas from people who don't know anything about how software is designed? Do you disagree? You easily can, but not based on the content, because it does not provide enough detail to prove whether they work or not.
Me: Recommendations opposed to plans, documentation, and formal processes in general.
You: No, they are not. The manifesto prohibits neither plans, nor documentation. Nor does it prevent it's implementation as a formal framework
There is a difference between "oppose" and "prohibit". I said "oppose". It does. People have misapplied this by doing even less than the authors probably intended. I say probably because, once again, the authors provided insufficient detail to know. For all I know, this could be exactly what they were going for. All I know for certain is that people claiming to follow Agile are doing an insufficient amount. Whether that was intended or not is not something I or you can prove. If I want to tell people they're doing something wrong, it is incumbent on me to explain what, in detail, they were doing wrong and how to do it differently. A slightly wordier version of "work better" is not sufficient in that regard, nor is saying that anyone who didn't improve was, by definition, not working better and thus that I'm not responsible for what my statement caused them to do.
There is still no silver bullet. Every magic formalised methodology tried so far has failed due to its tunnel vision or inflexibility.
A combination of experience, knowledge, flexibility and good sense will tend to succeed in a fairly decent way. You need to be able to zoom in on the details and zoom out and keep hold of the big picture throughout. You need to be able to design as well as build. Whatever valid criticisms the agile manifesto had of monolithic system development, it has been misinterpreted as a methodology that has reduced development to the crudest form of code monkeying, casting aside so much useful work that can be essential for a project to succeed.
Exactly - Agile is essentially a Cargo Cult like any other. The word methodology when applied to any project makes me want to start counting the spoons faster.
My yet to be patented non-methodology is:
1. Whoever's asked for whatever it is should state the key things they want from the system BEFORE anything is done and on a single A4 page (no cheating with using 6pt fonts or WingDings).
2. Get a few high-quality people (as few as is possible to do the work) to do the development in liaison with somebody/some people sensible (not too many or too few) with reasonable availability who knows what is actually wanted (see point 1).
3. Anybody who mentions the words "Stakeholder", "Agile", "DevOps" or "Waterfall" is offered a final cigarette before being taken outside and shot.
If followed properly the above usually results in something that works, on time and on budget in my eperience. Obviously this kind of approach really annoys people who use words such as "Stakeholder" which is why they need to be dealt with as above.
If you've never worked on a project big enought to need Devops that's fine, but unlike the other terms you list, once you're working on a serious project you will need Devops if you're ever going to get it deployed with a minimum amount of catastrophe.
Possibly the reason you're getting disagreement is that "devops" is another one of those words that a lot of people use but they all mean something different and everyone else doesn't know what it means in any given scenario. I've heard it used to describe the following things at least:
1. The same people build the application and manage the servers on which it runs.
2. The same people build the application and manage the environment in which it runs, but someone else runs the server. The dividing line between server and environment can be figured out at a later point, good luck.
3. Different teams manage these things, but they have frequent meetings.
4. The people managing servers should do it with IaC tools that are more similar to development. Who cares what the application developers do.
It's not surprising given these four that opinions on how good it is might vary widely. There's a reasonable chance as well that you have a fifth definition.
I've been in software for decades. The failure rate is astonishing - perhaps not so surprising since we're often working to build something new and untried, but of course we want to do better.
Agile is a way to organise a team, give them structure and a framework to hang requirements, work intake, revisions, development and testing off. As such, despite some management fantasies, it does not automatically deliver a working project, nor fix a broken team. If people don't want it to work, then yes, it really, really doesn't work. If people don't do all the other things, then yes, it really, really doesn't work. If people don't know what they're developing, or why then yes, it really, really doesn't work.
On the other hand, I've seen several 'impossible' projects deliver eight/nine figure returns in short time with tiny teams. Done well, with a good team, it really, really works.
The interesting thing about this survey is that it compares Agile with projects where the requirements were fully resolved before the project started. Unfortunately, that second thing is a pretty good indicator that management are switched on, the team is fully engaged and someone vaguely competent has had some serious discussions with all parties involved before the software is begun. So it may well be that what they are measuring is not the effectiveness of the methodology, but the commitment of the company to doing a good job in the first place.
> The interesting thing about this survey is that it compares Agile with projects where the requirements were fully resolved before the project started
The alternative is that the requirements are changing; but that requires an understand that "embrace change" does not mean "change is free".
>>On the other hand, I've seen several 'impossible' projects deliver eight/nine figure returns in short time with tiny teams. Done well, with a good team, it really, really works.
Yes because a small team has very little comms and management overhead - they can talk to each other.
The top person probably also trusts the colleagues not to need constant checking.
But a team of two or three in *separate* locations exponentially multiplies the coordination required and the possibilities for misunderstandings and borkage.
>> Yes because a small team has very little comms and management overhead - they can talk to each other.
You'll remember that agile recommends small teams. If you can't have small teams, you've not decomposed the problem well enough.
This is one of the consistent causes of failure of companies implementing agile - they pick and choose which bits they want to listen to, and assume that because they're "following the methodology" they don't have to put any further effort in. Those developers who most loudly criticise it often don't want to put the effort in to something they've not seen work for them, or modify their behaviour to benefit the wider team.
Having experienced a daily 'stand up' meeting of an agile team of over one hundred people, I can say with absolute certainty that the ability of management to misunderstand basic concepts when it suits them cannot be underestimated. Nor can the stubbornness of developers who don't feel they are being listened to.
"If you can't have small teams, you've not decomposed the problem well enough."
Very true and it would be interesting to know how many of the failed projects failed in an area tgat was hand-waved over in the original spec.
But if you choose the wrong decomposition then some of the small prblems will be insoluble (either literally or economically).
But you won't find that out until later, at which time you need to feed back all that you have learned into the decomposition problem and re-solve that.
But that decomposition is now necessarily harder/larger, because you now know several good reasons which the previously obvious decomposition isn't allowed.
tldr; Some problems are just Hard.
It also says you can't do something within a sprint then you've not decomposed the problem enough.
Sometimes there are big projects which require large teams and you can't neatly reduce every problem down to two weeks. Perhaps at that point the red flag should go up and someone should say Agile isn't suitable for these projects instead of persisting with it "because we are an Agile company now".
>> Sometimes there are big projects which require large teams and you can't neatly reduce every problem down to two weeks.
Hmmm.. I've worked for a lot of companies, from hyper-scale cloud computing down to single chip embedded devices. I am very suspicious of anyone who looks at a problem, strokes their beard and declares "I'll come back in a couple of months with a solution".
Big teams also usually mean there is next to no separation of concerns ("but it all depends on everything!"), and that somewhere an architecture astronaut has a big ball of string in their head. Turns out decomposition is also hard - and actually adds to the effort, not reduces it. But the point here is that agile doesn't magic away effort, it delivers projects that are more robust against change and often more maintainable.
"We need a big team that will work for months on a solution" can certainly deliver, but it also has some very nasty failure modes and often results in a code base that is horrendous. That experience was the driver behind the original agile manifesto and still holds true today.
This really depends what the criteria are for splitting a task. I can easily split lots of tasks into really tiny pieces, and each of them can be done quickly. I don't often do that because it just adds more work to manage tasks without providing any more feedback.
Consider as an example a project I have been building recently. I am the only programmer on it. In order to reach my last test version, I had to build lots of little components and link them together. It would be very easy to split the task into a list of components, then check each one off. The problem is that, when I have written two components and have seven to go, the code still can't produce a result that it couldn't before from the perspective of the users. Until all nine are written, the intended behavior doesn't work. I can finish each one in a short time, but I can't produce an observable improvement until they're all done.
But fine, we're not just looking at the user's perspective. What if this was a team project and people could see the results of finishing one component? It would work in this case, at least somewhat, because the components could be worked on in parallel. The problem is that, by developing multiple components, it would free me to change the design a bit to make improvements. If I split all of these and told other team members to do some of them, there might be clashes when they either didn't have the room to improve or where they proposed improvements that required us to delay our work to work on the interface. Not to mention that, for more complex software, there can also be tasks where the code is not of use to other developers until all parts are written and dividing it would be counterproductive. If it's just about reporting progress, those dividing lines can be set. If it's about having a new shippable version every sprint, there are tasks that are not served well by imposing that requirement.
Even if the project were to restrict itself to a "happy path" to fit into a sprint to get something everyone can see at the end of sprint 1, that means developers may have to leave several components unfinished. It may be as little as a front end with useless buttons and switches or it may be something more complete.
Later on developers have to go back to deal with error conditions, unexpected input, and so on and QA has to go back and test for errors and unexpected inputs and then test regression test the happy path again.
Meanwhile the project is under pressure to deliver because everyone saw it at the end of Sprint 1 but then no visible progress is being made. Or, even worse, the requirements aren't nailed down and everyone thinks they're a GUI expert and the next sprint is spent redoing the first sprint.
So really it's just picking at which part of project everyone is going to have to wait before seeing something.
Been a coder for almost 40 years. On a huge ongoing project I was on about 10 years ago, we adopted agile stuff. Managers stayed out of our way, and in progress large tasks stayed in waterfall mode. It worked quite well; far better than I image.
But I certainly agree that new projects, you need to truly understand the real requirements and deliverables. Current project is using MS Azure DevOps and it is nearly impossible to track related tasks, etc.
>> However, while the Agile Manifesto might have its problems, those stem more from its implementation rather than the principles themselves.
> Ah, the old "Agile failed because you're not doing it right". A classic.
Given the brief sample of code released, I don't think ICL/Fujitsu would know agile if it hit them over the head.
I think everywhere I've worked/every project I've worked on in the last 10+ years has said it's using agile - but I think only 1 ever actually was/did (and was successful).
Most "Agile" project are just ones where people have watched the first 2 minutes of a PowerPoint presentation and ended up as some half-assed effort where people pick and choose which practices they do/don't follow. Normally this ends up as "We're going to do the development Agile but everything else on the porject will still be Waterfall" - so you get no requirements in at the start, no business owner involvement until the end when they (eventually, about a month after you asked them to and 2 days before the go live) test, tell you it's wrong then complain about IT being crap, and zero documentation by anyone as "everything is self documenting". Oh, and of course managed by Waterfall PMs who's DCM isn't a quick round of max 12 people covering; what they did yesterday/what they are doing today/any blockers, but 30 people sitting in silance as the PM goes through every ticket in the Jira board asking for updates.
"Oh, and of course managed by Waterfall PMs who's DCM isn't a quick round of max 12 people covering; what they did yesterday/what they are doing today/any blockers, but 30 people sitting in silance as the PM goes through every ticket in the Jira board asking for updates."
Were you on my 9:30 as well?
Disclaimer: I don't know what the official definition of 'Agile' SW development is; I've just read a few of the XP books and thought them interesting.
TFA didn't state the definition -- if any -- used by the study to define 'success' and 'failure'.
One person quoted in the study talked about projects being done on time. One of the elements of XP (and 'Agile'?) is always having ~some~ working version of the code available for distribution and use, though it might not do everything everybody wants.
The spec is a human-politics battlefield, whether that spec is a formal document, frozen at a particular time (as in the waterfall model), or is embodied in "user stories" (as in XP). Whatever methodology is used, you can deliver a working system, which still sucks because nearly-everyone hates the way it works. But due to human-politics, you'll rarely get a system that everyone likes.
The original XP project (Chrysler Comprehensive Compensation System) and its eventual failure is an excellent example of what is wrong with Agile.
No, it is not stand ups or pairing or “gummy bares” or Kent Beck that made for initial success. It was a poor women who knew what needed to be done and who that spent tons of effort on providing requirements, who’s name nobody remembers today. The moment she left due to enormous stress of this, the project has failed.
I'm no dev and I'm a sysadmin by trade who happens to code, but agile seems liek "amateur hour on steriods". Agile just sounds like what every non-dev IT person when they actually sit down and code something useful. You bodge the code together most ly from old bits and some stuff scraped off forum posts, you get it mostly working OK, keep running it over and over to get the worst bugs out and put it use in prod ASAP. When it inevitably goes bang within the first week, botch in some fixes and then rerelease it back to prod, then go an fix test as an afterthought.
My experience of 'Agile development' is somewhat akin to drunken karaoke compared to, say Dame Shirley Bassey backed by the London Philharmonic Orchestra, both 'singing' 'Diamonds are Forever'. One is a polished poised performance with excellent diction, controlled power, the other a painful experience well worth avoiding.
"In highlighting the need to understand the requirements before development begins, the research charts a path between Agile purists and Waterfall advocates."
Get that spec. but understand that it can change because of, as well as during, code development. You need to be agile enough to asses possible updates to the spec and keep everyone on track to the latest agreed spec update. Requirements may change. The design flow is not some software package or holy software tome
I wish that were *always* true. I’ll mention the Abbott Freestyle Libre App disaster of last year.
https://malverntown-pcn.co.uk/news-mhc/new-issues-with-freestyle-librelink-app
It’s an app on your phone which connects to a wearable glucose sensor, which people with diabetes need to monitor their blood glucose. The failure was that Abbott updated their app on the IOS store, which then failed to connect to their sensors *in the U.K. only*, and then could not roll back to the previous working version because that’s not how IOS App Store works. At an instant, about 100k people suddenly became unable to self-manage their glucose and insulin, due to app auto-update.
The sensor itself is categorised as a Class IIb device, which IMHO is a fudge because although it doesn’t act on its own to inject insulin, its whole purpose is to provide people with the data they need to calculate that injectable dose. If that system goes titsup, you will either need to go straight to A&E or face safety of life issues within hours. Most sensor users haven’t used the old finger-prick device in years. Point is, the phone app is part of a safety-critical loop, and yet it isn’t part of any certification process. Abbott didn’t even seem to have any process to monitor the update rollout. The first Abbott knew of the problem was reading it in the news.
I’ve no idea whether Abbott was using Agile for their app development or not. But they certainly didn’t have a proper test or rollout process, and our certification system just isn’t built to oversee this kind of thing.
I genuinely don’t think so, although I have no inside knowledge. A couple other info points: the device is “so good” (life-changing) that until 2022 it had no equivalent competitors in the U.K. market. The *hardware* has about a 10% unit failure rate out-of-the-box, and when it fails and you’re out £55 for replacement from your own pocket, whether or not the NHS originally prescribed and you qualify for free prescriptions, for a device that lasts just two weeks. And people put up with that, because it’s still miles better than the old way.
I think the root of this, is that this is all so new that the companies involved just don’t know how to do it yet, their experience of process is in a different domain. Mostly, Abbott make drugs: high quality chemical engineering with controlled process in your own factory, they get tested, and ship them. “Medical devices” are still comparatively new to pharma companies, but they are mostly mechanical and materials devices. Electronics, *consumer* electronics?! that get implanted by patients in their own homes, that’s wild stuff. *Software* running on devices that they don’t make?!! That’s totally outside decades of core experience. I suspect that Abbott will have outsourced the app entirely, to people who “know how to write apps”, on the basis that the regulatory classification is that it isn’t critical software.
It isn't, though, because it strongly encourages cutting tasks until they fit into a single sprint.
The number one cause of technical debt is shortening the timeline so there's not enough time to do it properly before the deadline.
Squashing a 3-week problem into a 2-week sprint creates technical debt. Even if you tell people "don't worry if it rolls into the next sprint", there is always that push to cut a few corners, ignore a few edge cases, skip a few tests...
Don't forget that this 'Agile is rubbish' survey was for a book extolling the benefits of a methodology that "offers ... proven strategies for successful personal and business transformations" (quote from the book's website). A book produced by the very same organisation that commissioned the study in the first place.
Also, I wonder if the selection of Agile and a project's failure are down the same reasons, rather than the former driving the latter. ie if a project looks doomed from the start, someone might be more inclined to use Agile as a futile attempt to save it.
I agree with you there. For all the problems I've discussed about Agile, I'm not convinced that this overwhelming condemnation of it is accurate. I'm pretty convinced that there are at least a hundred different processes being called Agile, so there's no way they can be accurately averaged and compared to anything else.
The UK Post Office's infamous Horizon IT system stands as a cautionary tale for large-scale projects. The project demonstrably lacked key Agile principles. Firstly, it failed to deliver "Working Software." Secondly, the focus seemed to be on "process and tools" rather than "Customer collaboration." Subpostmasters, the system's true users, reported issues that were dismissed, leading to some even facing jail time. This is the antithesis of effective customer collaboration.
Agile emphasizes business-to-user collaboration, not business-to-business. In Horizon's case, the "customers" were the actors/personas using the system, like Subpostmasters, Support, Accounts, and Managers. The project failed to provide them with the necessary tools for efficient work.
In my experience, project failure often stems from a lack of alignment between business and development teams. Agreed-upon goals are crucial. Additionally, engineering capabilities need to be sufficient for the task. As the Agile manifesto states, "We are uncovering better ways of developing software by doing it and helping others do it." However, this requires developers to:
Create functional software that meets user needs. In my view, "working code" should be "self-testing code." This promotes clear documentation and understanding of functionality. Businesses should be able to review tests and confirm intended behavior. New tests should be created to expose and fix issues as they arise.
By incorporating these principles, future projects can avoid the pitfalls exemplified by the Horizon system.
This post has been deleted by its author
I know of one team tasked to produce something for a hospital based on vague wish lists, prototyped a solution and demoed it to the customer. They spoke to nurses on the ward who said
"this doesnt work for us. It does not have the patient data we need. We need these fields.... and these fields should be red if ..."
"where are the lab results? What do you mean.. logon to another system to look at them. Why cant I click a button and you just display them"
"Why does each nurse have to logon and off each time. We need a userid per ward which is always logged on - but provides screen lock"
They went around the loop, and found the nurses wanted a screen which connected other systems together without having to logon/logoff to other systems. This was not covered by the original requirements.
They threw away the prototype code and started afresh. Half way through the nurses were very happy on the progress. Then the security people said that every nurse needed to logon/logoff for every interaction so they could track who had accessed data. So nurses spent a lot of time logging on and off!
And they started designing it all over again!
I bet they didn't need to logoff, requiring a full re-authentication to get back on. I bet the risk here is simply a nurse putting down the device for a moment.
This means it would have sufficed just to lock the session with unlocking being facilitated by no more than swiping the correct (ie, the logged in user's) lanyard.
But to arrive at that design you need to have an end-user to tell you what's impractical, a security/legal person to tell you what you can get away with, and a developer to tell you what's possible.
And they all need to actually like each other enough to negotiate a solution.
ColinPa: "Then the security people said"
Quite likely that GDPR (or DPA) requires that any individual accessing Personal Identifiable Data of a patient has to be uniquely identifiable themselves. This may be a case of the national law not taking account of older, efficient, working practices. When patients' data was held on paper records, there would not have been any recording of who gained access when to the same level of detail computer system with accounting is capable of. In the UK a Caldecott Guardian reviews access to medical records by anyone without a proper relationship with the patient.
So, please do not blame 'the security people' for remembering the law. And, of course, detailed records were essential for catching the murderer Lucy Letby (https://en.wikipedia.org/wiki/Lucy_Letby).
Agile, for when failure is an / the option.
After watching several agile implementations and subsequent projects collapse, I have yet to experience the wonders and miracles of a successful one. I have also not yet seen a unicorn, but I am certain they both exist /s
and break things has a whole new meaning when it comes to industrial controls....... usally moving fast because something has broken.......
3lb casting in the face anyone :)
<<has spent a trying afternoon trying to debug the handover/cut off code on one of the twin spindles because someone changed the standard macro we use...
Article claims that Fujitsu's Horizon system was 'developed' using 'agile' process.
The inquiry is still in the news:
"Former Post Office chair Alice Perkins was warned about potential faults in the Horizon IT system as early as 2011, an inquiry has heard.
At the time, the Post Office had prosecuted hundreds of sub-postmasters for fraud on the strength of faulty data from Horizon accounting software.
It would carry on with these cases until 2015.
Ms Perkins, who was giving evidence at an inquiry into the scandal, said at the time she did not make a link between the two.
She was also told that the Post Office had "driven a very hard bargain" on the price of Horizon, and in return developer Fujitsu had cut corners on the quality of the software."
From: https://www.bbc.co.uk/news/articles/c88847v14e3o
As exICL (retired 30+ years ago) who had nothing to do with Horizon but is belatedly trying to follow the inquiry, I have questions regarding the relationship between PO FJ and the spms.
Were the essential processes needed by PO to support in place, with resourcing at the level needed by the SPMs from day 1.
Was the PO structure modified to cope with an online system where all units had to be aware of issues happening all over the country
If senior management of PO were not aware of their staff prosecuting SPMs, how could they ensure that these staff were working to standards set by the PO.
Did FJ understand the legal requirements of Expert Witnesses as apart from Technical experts.
Did FJ compromise their deliverables in order to meet timescales driven by politicians (and or Royal Mail Group) looking for quick results
Etcetera
I know nothing of Agile but software projects have deliverables which are time related, needing to be accurate from day 1, have to be fully defined to produce acceptable results. I would argue that Horizon has to be correct in order to allow the production of acceptable auditable annual accounts on time to meet financial timescales. This is not imho compatible with Agile with it's try, throw away, and try again attitude
(1) I've done many process documentation projects. Every single time it turned out that the people executing the process knew almost nothing about the people up or down the process chain.
Indeed: Quote: "We throw away what they send us." Quote:"We have to completely rework what they send us." Quote from director: "I did not know that we had two completely separate processes for X."
(2) User Stories: "So we have all these yellow stickies on the wall."
(3) Product Owner: "Never seen them!"
(4) Planning: "It turned out that we needed an architecture sprint before all the other sprints.....and it never occured."
So......four random personal observations..........
"What even is Agile?" - you ask 4 people and get 6 different answers.
A solution that helped for some, not all, projects in the past: do the design 'waterfall', and build "agile" (what better to say is 'scrum', etc.)
There is no agile if there's a time box, money box or resource box around your project, especially when the budget holder says 'i want it all!'
I'm sure my esteemed commentards have said so before i write this post.
I'm that curious sucker.
The book is about 141 pages of cutesy fluff "telling a story", the remaining 3 pages mostly consisting of "Technique Summarised" are little more than found in these comments.
"Define Problem", "Find Solution", "Implement Solution", "Analyse and Repeat previous."
i.e. Classic Software development spiral model
+ 12 tips .
I have worked in a team full agile on a very complex requirement. We delivered a working product.
I have also worked on plenty of ‘agile’ projects that…. May not have gone so well.
Two key differences on the successful agile project:
- all team members went on the same agile course at the start of the project and therefore had the same understanding of what agile is
- the team was properly resourced with embedded SME’s (who were not also doing bau work), an experienced scrum master, BA’s, tester and devs all sitting together and working as one team.
By comparison, most ‘agile’ projects are the same old same old with a smattering of ‘agile’ words to please the boss.
Somewhere between the two extremes is a pretty good set of concepts for success. You need a clear business plan, you need inter-project dependencies managed, and you need rapid client feedback. Treat Agile or Waterfall as hard rules and you'll have a hard time.
That said, sometimes a project fails because feedback is really bad and you're relieved that you didn't bother finishing.
LOL - Measuring an Agile projects on waterfall metrics will definitely not work.
Scope and time...
Agile is slower, it is incrimental and the benifit is not on closing a project out on budget. It is delivering value, most of the time the 'original' requirements is way off and delivering that misses the mark completely.
Agile self organizes to the best local optima (if directed by customer centric feedback cycles) it often looks less complete because everything is in a constant state of improvement with only the real important aspects evolving forward.
Is the goal a 12month signed of project with zero users or a revenue generating product that is still being worked on 24months later but had their first paying customers 6 months in?
If worked on many Agile projects and none of them were agile - bar one. The one where I was CTO and we were agile. We increased user count by two orders of magnitude and transformed the company into a massive success.
Agile is not a methodology or a process or a practice. It is something you are (based on the dictionary definition of agile). It is based on understanding that software creation is not a repeatable manufacturing process but a R&D process where you are gaining knowledge. The manifesto and its principles merely highlights some of the things that flow out of pragmatically accepting these facts.
Agile is far and away the best approach I've seen to creating software. Please don't confuse your experience of the Agile Industrial Complex, with real agile. Scrum, SAFe etc is not agile.
See https://martinfowler.com/articles/newMethodology.html for a primer (ignore the stuff at the end on the implementations nobody uses any more, it's a very old document).
This post has been deleted by its author
I'm out of the business now, safely retired. But I had 35 years of fun working mostly on my own. I got so much done without tedious project managers getting in the way.
However, in the latter stages of my career, I had to suffer the pain of agile and scrum. It was awful. A thorny problem would come along and I, as always, relished the thought of dreaming up an insightful elegant solution or approach. But, no, a directive on high would come to just bang something out every two weeks, fitting all the other tedious agile work around it. It made me so miserable, it was the final nail in the coffin for me.
Queue Martin Fowler's State of Agile Software in 2018 talk right now! All these comments prove to me Agile is highly misunderstood.
TL;DR: Maybe instead of blaming your failure on all the principles you violated, you should blame your incompetence! If your "agile" project fails, you were not Agile.
Agile is a set of principles, NOT a project management process or methodology. Scrum is NOT Agile. Kanban is NOT Agile. Agile is 12 simple principles. Apply those principles well, you'll be successful.
For example, one is having self-organizing teams, meaning the teams decide the best process to get the work done. If your "agile" project is failing, then you didn't organize well, did you? And if you noticed you were failing and didn't adjust your processes, you violated the self-reflection/tuning principle too.
Something else that doesn't align in my head is the methodology for choosing what constitutes a "failure" or a "late" delivery. Interestingly enough, the Agile principles NEVER discuss estimating, timelines, etc. And since the primary measure of progress in Agile is working software, how do you manage to never have anything working along the way and expect on-time and successful delivery?
As to testing, Agile focuses on working software. How do you know it's working if you don't test it in one way or another? Attributing any excuse for poor testing to Agile is simply an excuse for unprofessional work. Bad testing practices are an organizational problem, not an Agile problem.
Finally, I have seen many organizations hire contractors that don't care about what they are building and only do what's on the task to collect a pay check. As such they don't ask questions, they don't see the Big Picture and they don't care about quality. How do organizations expect a good ROI from these hiring decisions? This violates the "motivated individuals" Agile principle.
So, you thought Agile was something it wasn't, you stuck the agile label on your project without knowing what the meant, things failed and you blame the principles you didn't follow? Seriously?
And yes, I have seen great successes throughout my software development career from a proper application of Agile principles. I wouldn't do it any other way. All failures I've seen come from several violations to the Agile principles.
"TL;DR: Maybe instead of blaming your failure on all the principles you violated, you should blame your incompetence! If your "agile" project fails, you were not Agile."
Are you trying to make people hate Agile? If not, the sentences I quoted were a very bad idea.
A lot of failures in project management are not the fault of the Agile philosophy per se, even when it is used to justify the bad decisions. That doesn't make the principles blameless for the problems they can and do cause. You have attempted to elevate them to a position of faultlessness which nothing can live up to. You have further demonstrated that you are unwilling to consider that there may be any problems whatsoever with them. The rest of the world knows that, even if it's something they think is pretty good, that everything has flaws somewhere and will only improve if those flaws are identified and corrected.
If you want to reply to people pointing out what they see as flaws and explain why, in your opinion, they are not flaws, they aren't related to Agile, or they aren't as bad as we think they are, we might get somewhere. If you reply to every flaw and say that, although we've drawn a direct link from it to something core to Agile, it's totally unrelated but you can't be bothered to explain why, then nobody will be convinced. Here are some examples where you have misconstrued points and ended up in error.
since the primary measure of progress in Agile is working software, how do you manage to never have anything working along the way and expect on-time and successful delivery?
Not what happens. People have working stuff along the way. The customer wants something different, but there's not enough time to change from what they said they wanted to what they actually want. They will either get what they wanted late or they'll get not exactly what they want on time.
For example, one is having self-organizing teams, meaning the teams decide the best process to get the work done. If your "agile" project is failing, then you didn't organize well, did you?
So that's the only reason in your mind that a project might fail? I would think that it's pretty obvious that lots of other things could cause project failure. As an extreme example, if an asteroid hits an office building and wipes out some teams, then your organization is not at fault for the productivity decline, is it? As a more realistic example, if you've organized to build something which gets changed late in the process, it is not the fault of those who built what they have now that the final product concept is different and isn't ready.
... and in a certain range of projects, mostly small. Published in 2001, but reflecting the environment mostly from the previous ten years or so. The whole software environment is so different now in pretty much every way, and projects average larger, and web-scale stuff always has crazy considerations, and the "stack" now is ten times what it was then, etc. And Scrum is not really Agile or agile. Basically Agile/Scrum projects fail on day one and then see how long they can bumble along, but the thing is they get something right, too, which is getting that MVP out there soonest.
So yes, but the Agile/Scrum failure rate is basically 100%.
And then there's the contribution from "Extreme Programming" which goes beyond the original manifesto but contributed a lot to the Agile/Scrum environment, and again it made sense at a certain time and place, 20-30 years ago, but less and less as time went on, but also gets some things right and others not so much.
But then what is the right answer? I dunno. Changes every few years, with the industry. Speaking of which we all were talking and doing "spiral" or "iterative" development twenty+ years *before* the Manifesto. So it goes.
The issue isn't Agile. The issue is the distinction and separation of project management from software development, which the Agile Manifesto is about -- essentially that programmers know enough about how to work that dealing with PMs is not necessary.
The real issue is that PMs ate the Agile Manifesto and took it all the wrong ways, deciding that "Agile" means "don't figure it out first." Brrt, wrong. Within Agile, you're still supposed to do requirements gathering, the difference is that it recognizes that sometimes exploratory development is crucial to finalizing requirements and sometimes the requirements will have to change.
The projects flopped because they used "Agile" as an excuse to not gather requirements first.
YES! And that having all the all the requirements (like the exact colors we want to use) doesn't have to happen to start coding. And I think that's the real problem: everyone hears what they want to hear when they hear the word agile: "I can write bad code that I'll fix later." "I don't have to write tests." "I don't have to figure out what we are doing now, I can do it later." "I don't need to document anything." "I am guaranteed success even though I don't know anything about the project I am managing."
Having been in quite a few "Agile" projects over the past decade or so I have found the MAIN problem is the customer/end user and their requirements.
Easy example: Finding the current state of a process to analyze it for the new/improved (mutually exclusive terms btw) process.
What they SAY they currently do, what they ACTUALLY do, and what they are SUPPOSED to be doing are 3 VASTLY different things. Add into the mix years of "I have been doing it this way for decades for reasons LONG forgotten" and of course the "I'm retiring so I'm only going to teach the new kid WHAT to do and not bother to explain WHY I'm doing it" and then the poor new kid is thrown in with the poor BA (who is usually an outside hire specifically for the project and who has NO clue about the actual business) .
You know you're in trouble when your code monkey (That would be ME) gets the "specification document" and has to go back to the user and BA and tell them "But what about XYZ scenario" which completely blows the current solution out of the water. Of course this only happens if your developer has been around a while and actually has a fairly good understanding of the business and its processes (usually much to his/her disgust). If you have a bunch of relatively new/young hot shot developers, they will dive headfirst into the code based on the specs, deliver what is asked for and then get my favorite quote of all time "Yes, this is what we asked for, but it is not what we want".
Personally I like the "horses for courses" approach. Regular feedback is important. Daily standups (no more than 15 minutes), while a bit of a pain do add value in the sense that everybody is now aware of where we are/what any holdups are BEFORE they become mission critical. MOST of the other "ceremonies" in agile are just a waste of time. But if you don't have a clear and DEFINED endpoint in mind then working rapidly towards that ever moving goal is just going to cause frustration.
My 2c (Given the current exchange rate, worth even less)
I suspect that management want a silver bullet. If there are methods and procedures and all that then they feel they have a measure of control.
Back in the day, I was tech lead on a bit of niche shrink-wrapped software. This was when we had CDs pressed and manuals printed (on paper) so whatever shipped had to be right as there was no downloading a patch. Because we were highly profitable, management didn't interfere too much. We had a work tracking system (implemented in Notes!) but nothing in the way of rituals. When people needed to talk to each other, they did. The expert users were on hand all the time - we even had a couple sit in with the dev team. That made changes really fast. We had a QA guy who was a colossal PITA but ensured that the damn thing was tested. He ran his regression suite with a coverage analyser to make sure he was hitting every single line of code.
We did do a build every couple of weeks for the expert users to test. Shipped a release every 9 - 12 months. My feeling was that it worked well because 1) We had competent people 2) We had deep involvement from the SMEs 3) We designed our method of working around what worked best rather than slavishly following a method.
Other stuff I have been on which has not worked so well showed opposite trends - little or no user involvement, massive bureaucracy for the sake of it, "agile" is something IT do - it doesn't affect the rest of the company. Anecdata, I know.
"This was when we had CDs pressed and manuals printed (on paper) so whatever shipped had to be right as there was no downloading a patch."
We were making patches available as soon as data/code on High Sierra (followed almost immediately by ISO 9660) formatted CDs was available, in the mid 1980s. Remember, some of us had been using FTP to get patches for about a decade and a half at that point.
"Did you have to create a few thousand CDs and send them to 40 different countries?"
Well, we started with tapes and much smaller numbers, but eventually, after CDs were invented and things got wider exposure, yes. And then TehIntarWebTubes became more accessible and we shifted that part of things to Walnut Creek CDROM ... The patches were still on our FTP site, though. And mirrors near you, probably.
Send your request and an SASE with blank media of your choice to CSRG c/o the University of California, Berkeley ...
There is no such path, waterfall is embedded in agile, it has to be, https://www.dmossesq.com/2014/01/what-are-agilists-talking-about.html
"Waterfall can also be slow and costly, with changes challenging to implement" – agile can also be slow and costly, with changes challenging to implement.
How many of those projects actually used Agile, though? Call it "Agile" because that ticks a box, gets staff in, gives a professional look, and gives the CTO something to say to VC -- but at the same time, keep using a spreadsheet, and when push comes to shove, abandon the plan, shout a lot, and make the staff work the weekend.
I was working as a contract test analyst on a project for a major British insurer. In one of the "Scrum" meetings I pointed out a bug that was causing several areas of failure
The team leader ("Scrum master") said something to the effect of "That's not in the plan so we're not addressing it"
I pointed out that we were supposed to be Agile.
His response was priceless...."We're not THAT Agile"
I duly made a diary note of his response...6 months later when asked why so many bugs were arising in several disparate areas, I pointed out that this one bug was probably the root cause.
SM: "Well why didn't you test and raise a major bug then?
Me: You told me not to"
SM: (Indignantly) "When did I ever say that?"
Me: (Turning to relevant page in diary) "In the "Scrum" meeting on <date>
There's a moral or two here, but I'm not sure what it / they is /.are.
Talking about frameworks appearrs to be outside the scope of the article. More specifically, the article mentioned the AGILE (NOT SCRUM SPECIFICALLY) IS RESPONSIBLE FOR PROJECT FAILURE.
I would argue that agile can work in projects but would be better suited to complex and dynamic environments. If your use case is dynamic, you need a dynamic mindset approach to tackle the problems in that space.
Mentioning scrum in this context is reductionist.
Some good ideas have come up in and around the Agile movement. We take a shopping list approach:
- Stand-ups? We tried that and found it useful to us (especially as we all work remotely).
- Nightly builds? How the hell did we survive without those?
- Introduction of frequent automated testing? Yep.
- Writing simple specifications early in Gherkin*? Yep.
- Getting rid of the test team? Erm, nope.
- Scrum and sprints? You've got to be fucking joking. Who'd want to put themselves through that circle of hell?
Oddly enough, our proposals still state we use a modified waterfall model because no one can be arsed calling it anything else.
* Still end up with proper specs to sign off later though.
Perhaps this is why I spend all my time at work dealing with the effects of crappy software.
When I aged out of the software development racket after 20 some years on Wall Street I decided to give technical writing a try. I was pretty good a writing documentation and even enjoyed it sometimes. After getting a certificate in Technical Writing from UC Riverside a went out in the world to shop my new qualifications. Imagine my dissapointment when everyone I talked to told me that nobody does documentation any more. That's why I am ending my working life as the systems manager/network administrator/all-around IT guy for a small (30 users) wahehousing outfit.
......is the "regular delivery of working software" thing....
After all, this probably means that the customer for the software is forced into the "regular implementation of process change".
I'm sure there are customers out there with a taste for regular re-training and regular re-implementation.......but I've never met them!
It can be very useful if the conditions match, specifically that the software has a lot of user*-facing features and the user is actively working with you to get it. The point is that, although they've specified a new thing and you say they'll get it in a year, you can actually show them parts of it every month or so and get their feedback. If they want something developed in the first month to be different, you can change it in the second month. That works badly if you deliver the whole thing in ten months and they decide they want it changed. In the latter case, you usually have to refuse to change it so they can have their software on time, whereas if you can change it quickly, they can get something closer to what they want and still on time.
The problem comes when the user isn't checking on these things. Every month, you send the next development version to a manager who doesn't use it, the manager forwards the email to someone who either doesn't use it either or does, but they're too busy using the old version to help check on the new version, and the developers think everyone's fine with it when nobody's looked. Doing this well effectively requires that the user has an internal beta team, not to catch massive bugs (although they might and that is helpful too), but to report where they would like things to be designed differently. If they are unwilling or unable to do this, then this method no longer works and the developers need to use a different method to ensure their design matches the requirements.
* User in this case may be internal or external, and it may be at various levels. Other programmers can also be the user. In fact, that sometimes works better because hopefully they understand the importance of testing incomplete versions.
I thought the point of Agile development was to fail fast and often. Although this implies it just fails.
I've been told a few times that agile development is a perfectly valid methodology if done properly.
But IMHE it's always been an excuse to quickly chuck out a half finished pile of shit. Often never to be finished.
Throwing that 'something' into production is normally always accompanied by a fanfare and back slapping about how fast and cheap it was done.
Reading the nearly 200 comments I am glad I avoided the software development side of the trade for the defenestration side. ;)
Agile is if you can get through the office door before being pitched out the accidentally open window. :)
Today I saw medical interpreters looking through their notes (minutes?) of the patient's examination etc. What surprised me was that those notes were in an A5 notebook with carbons. I assume the top copy goes in the patient's record.
I often wonder whether our use of IT blinds us to the actual effectiveness of the processes it replaced.
Whatever software development method(s) are actually being used in the real world the aggregate results are a pretty convincing statement they are all rather rubbish.
"I often wonder whether our use of IT blinds us to the actual effectiveness of the processes it replaced."
Probably, but at least part of that is due to users who use the familiar workflow rather than the better one. Yes, they may be able to demonstrate that the real problem is the software not doing something it needs to, but in other cases, the software can do things but someone won't learn it. Not every system that didn't get switched is because the software isn't fit for purpose. Inertia is a big factor, and there are various things we can do about it, from training to more information about why this improves things. If we assume every failure is a quality problem, we may be punishing ourselves for the wrong thing when what was needed was a better introductory training on the new option. Even if it is a quality thing, it could be due to bad programming, good programming of a bad design, or good design of a bad process, and you won't fix the situation without figuring out which.
Some hilarious comments here from people who clearly have no real experience with effective agile. Agile is dead simple, it isn't all the BS, it's just about iterating rapidly based on actual user feedback. All the methodologies beyond this are just trimmings & consultant fodder. Almost all agile failures are from the org not really iterating or basing their iteration from fantasy, rather than real world feedback.
I challenge anyone to say that a 6m "requirements analysis" phase followed by 12m of "build & test" then a "release" will produce more total value to customers than 18m of direct test and learn iteration. Real world experience > on paper specs and silo'd dev.
Crux of this is not agile. It's product vs project. One is right, other is dumb.
Problem is that everybody seems to read and remember only the first half of agile manifesto. It says documentation is important but I kepp seeing so many people who interpret 'not as important as working code' as 'not important at all'...
It's convenient, but ultimately stupid.
I think it's the truth.
I am an agile enthusiast, but applying to the letter the agile principles can lead to disasters.
I led a project in 2014 using Scrum, and as the architecture was well understood, the project went as planned.
However, in 2016 an infrastructure replacement project started without a full investigation of the existing and of the new architecture. Result: after 10 months in the project, the architect realised that the APIs he had designed were not fit for purpose, with other parts of the old system still not well understood.
So, I think that an initial phase is needed - especially on replacement projects - to understand the old system architecture, the new architecture and how it fits within the other existing systems.
WW Royce is the man whose name is always associated with the waterfall method, the waterfall method is always lampooned by the bien-pensants and so WW Royce must have been an idiot. Such an idiot that he ran IT for Lockheed. And guess what his job was before that – https://www.dmossesq.com/2014/03/waterfall-wanderers-0-0-agile-athletic.html
Over the last 20 years "agile" has come to mean any number of nonsense things, none of which lead to agility, and many of which lead to tears before bedtime. Time and again organizations adopt some bastardised form of scrum with certified agile practitioners to help them. Far from not creating requirements before starting the project, requirements are defined in advance, the end point is fixed, and then a dev team go through this charade of fixed length sprints of between 2 and 4 weeks long.
Stories are not broken down into small enough pieces, the room to change direction is limited, and the delivery increments are too big. The methodology pushed on the dev teams is too restrictive, retros aren't taken seriously, and (one of the things identified by this study), there isn't real psychological safety so that teams can challenge practices or directions that are failing.
See this article for more: https://tanzu.vmware.com/content/blog/why-your-agile-initiative-is-failing
As someone pointed out above, it has become a cargo cult. And don't get me started on SAFe.
I still harbour the strong suspicion that if you have good people (tech and user side) working on a project, they will succeed - despite any methodology you try to impose. And if you have crap people, they will fuck up - despite any methodology you try to impose.
Since all this garbage came out back in the early 2000s with claims of faster development times and whatnot, I have been writing and arguing against such new processes for years.
There is absolutely no such development concept that can work effectively without the so-called "waterfall" model, which for all intents and purposes, merely demonstrates a logical way to go about developing a project. There are many variations on the "waterfall" model but none have to date been able to eliminate it... until Agile came along...
And now we are seeing the results.
If you want a project to be completed on time and within budget then you follow the standard software engineering practices for project development and implementation.
If you don't, like the majority of technical managers who still believe they are geniuses, then use Agile...
Steve Naidamast
Sr. Software Engineer
Like other people commented above - agile is meant to address a problem of building the wrong thing and not realizing it until a lot of money and time has been spent.
"Within budget and on time" as a success criteria is a bit absurd because it implies that you know what you are building upfront to a great detail. I'd argue only a minority of IT projects need and can be done that way. That's when you have other methodologies designed to optimize for end-to-end predictability of estimations and costs at the cost of the speed of development and pivoting (used in aerospace in other systems where cost of defects and changes are high).
How quickly people forget the times when companies would build software for months before releasing it and realizing all the things they did were unnecessary, wrong or no longer relevant.
Also, as someone who works at a big "agile" company I estimate that 90% of all teams which claim that they follow Agile are doing the "Cargo cult" of Agile i.e. standups, sprints and other motions without actually following the spirit. How many of those teams are cross-functional? How many of them have customer or customer representative on them? How many of them communicate and collaborate on their work? I don't think there would be many of those.
So yeah, pick a non-relevant success criteria, apply it to a catchy methodology which nobody actually follows and then claim it fails to achieve that criteria. What is it if not trolling?
I imagine some of those tagged as CentOS 7 are used for scientific computing. Fairly common for organisations that have some sort of RH subscription to run HPC cluster with CentOS with critical services running on RHEL hosts. Mostly saves expense.
The Springdale scientific software repos are often layered on top of the *EL7 hosts. Princeton was rebuilding its own distro but appears to have stopped at 8.8 and 9.2 around May 2023.
A question I would have is whether an organisation that has a RH subscription and therefore legitimate access to the RH sources is prevented from producing its own rebuild for internal use?
I am guessing deep within the licensing that you can but your rebuild would be audited as a chargeable RHEL install. A bit like Oracle and Java - Big Blue was up to these shenanigans long before Big Red was as twinkle...
I run AL8.10 on my own kit having upgraded from RHEL7 (dev subscription) and CentOS7 - no real dramas apart from the disk array's SAS HBA being blacklisted by RH mpt3sas kmod (AL now undeprecates this hardware.)
If Almalinux were to fold I wouldn't have too many problems or qualms moving to SuSE.
This reads like someone who had their feelings hurt they weren't allowed to cowboy code, so they ran screaming into the void with barely an understanding of what they were talking about.
1. No one starts an Agile project without some level of planning and requirements.
2. Agile is the methodology supported by the manifesto and principles. Agile is made up of 20+ frameworks depending on who you ask - and some of those frameworks include developer centric principles and processes. A blanket statement that 'Agile' is the problem...well again...barely an understanding...
3. Can someone show me where a line of the Agile manifesto or any of the 12 principles say "do not plan"?
4. If you've experienced bad Agile...thank a leader for going with the big consultancy who brings in the same playbook whether you're 300 or 3,000 employees. Thank them for choosing tools prior to defining processes within the context of your company - so by the time you realize it's the wrong tool, you're too entrenched to change it without political warfare. Thank them for not getting you the right training to help with the transition - we all know the whole engineering department doesn't want to be Certified Scrum Masters. And of course thank them for having not prepared your company culture to handle a methodology that thrives when there is psychological safety.
Anyone who has been doing real software projects knows that getting complete specs and implementing them in a predetermined timetable is a total fantasy.
You need flexibility, and Agile recognises that. But swinging too far in the other direction does not work either, except for very small projects.
Any project that produces working results is in reality a mix of "waterfall" and "agile", no matter how it is officially presented.
Agile works. But Agile doesn't mean that sales people (including the CEO) can keep shifting the goal posts every week. Agile doesn't mean fixing the delivery date before you know what you are building. Agile doesn't mean no design or tests. No process can succeed under such circumstances. You need a north star that doesn't move and you need customers who are willing to engage in incremental requirements discovery, implementation and delivery. At its heart, Agile is about building stuff that works in the real world, for real people.
Not sure how "Our research has shown that what matters when it comes to delivering high-quality software on time and within budget is a robust requirements engineering process and having the psychological safety to discuss and solve problems when they emerge, whilst taking steps to prevent developer burnout. This is fundamental to the philosophy of Impact Engineering." - could not be used within an agile framework (the quote is directly from EngPrax's press release).
Waterfall tends to keep users and product owners away from the developers during the project leading to the well know 5 stages of UAT (Disbelief, Anger, Bargining, Despair and finally the Acceptance) - Agile keeps the users and product owners closer (so fewer surprises come UAT time) but this can lead to other issues. I found the most fundemantal factor that determines a projects success or failure is down to unrealistic expectations from higher management on timescales and costs.
The agile manifesto doesn't actually mean there should be *no* requirements and *no* documentation and *no* well-defined functionality to test against... but this is somehow the default position of the current developer generation that came of age after the manifesto took hold, not having had experience of what the productivity of an actually well run software project can be like.
For me, the agile projects that worked well were so far all populated with fairly old-school types that may even have been enthusiastic about agile, trying to give developers a lot of power, but at the same time kept quietly track of the overall project in ways they had found useful in their previous "non-agile" projects.
And in the current developer generation's mind, anything more is immediately considered "waterfall" (which itself has been strawmanned to supposedly having been the "half a year of requirements specification, half a year of implementation, half a year of testing, then first contact with customer" model).
Meanwhile, my ancient experience of the pre-agile software development project model was, generally speaking, "a well-defined project that still works in an iterative way, checking against customer input early and often" (with those experienced project managers who guarded against the perils of waterfall in their own hard-won ways). Note: capturing the actual customer requirements is in some ways easier when you have your initial set of requirements already available to be confirmed, or more frequently, shot down. Call the initial requirement set your "product vision" or whatever - as long as ego and inertia do not get in the way of changes to fulfill the real world needs when they become evident, the iterative-but-requirements-driven projects can be surprisingly nimble when pivoting.
In 2005 I had a project for which all known systems and methodologies of Project Management are entirely useless ... and sometimes you have to do what is necessary to get the job done.
In September 2005, I received a phone call, to the effect that my (now late) wife's father had been taken ill. In Russia.
It turned out that he had been chasing women on the Internet (at the age of 81!) and he had had a stroke. He was in a hospital in a town called Komsomolsk-na-Amure (Komsomolsk on the Amur River) in the Russian Far East (nearest borders: China, North Korea, or over the water to Japan). His lady friend (herself in her late 70s) spoke only marginal English, and her friend down the corridor was translating for her.
Your task, should you accept it, is to project-manage him back home, alive and within a sensible budget.
Scrum, Agile, PRINCE and others may be found entirely useless. And you can produce as many reports (charts, checklists, presentations ...) as you like, but there is no-one to report to. Likewise, you will be the only team member attending any meetings you hold.
For the record: I succeeded [long story...]. How about you?
If it involved smuggling and spiriting him out of the country, gut feeling is your solution involved a couple of fishing boats and landing the gent on northern Hokkaido. (The fishermen up there don't give a shit about international politics or so-called national borders).
If it was all above-board, with passport & etc, there are any number of not-for-profit organizations which exist precisely for this reason, usually at no cost to the family.
Regardless, hopefully he had a decent late innings.
And if one doesn't bother to read the counterarguments, one fails to convince anyone that one is correct after all. We've had the "if it didn't work, you didn't apply the magic right" people already.
A lot of the comments here point out actual objections to Agile, as in limited issues that apply in particular situations. Maybe you could explain why those are incorrect? Crazy as it seems, a lot of us have read the manifesto and the list of principles and have a real basis for our critiques. I think many of the bad uses that claim to be Agile are doing things that you and the other authors did not intend. That is not the case for all of the bad applications. Even when it does apply, that doesn't automatically mean they weren't doing Agile. There are many things you wouldn't like which your manifesto does not recommend against.
Agile is a series of practises best suited for dynamic environments of varying degrees of complexity.
It is not a thing you deliver on a date. Rather it is a culture that enables and fosters innovation.
Various frameworks can help foster these practises but it has to be worked at continuously.
The practises should be enabling a mindset of
- Continuous learning
- Adaptation
- Openness
- Self organisation AND
- Collaboration
Agile like waterfall doesn't guarantee anything. You are more likely to be successful however if you are constantly learning and adapting.
So as a remedy to the dysfunctions highlighted in the article, focus on enabling your teams to have grow an agile mindset and foster an enabling culture especially if you are on the senior management board or in a privileged position of influence.
https://www.adventureswithagile.com/what-is-the-agile-mindset/
https://www.bitesizelearning.co.uk/resources/five-dysfunctions-of-a-team-summary-pyramid
Don't wait for agile to be done to you. Be AGILE!!!!
Never believed in waterfall or agile being the only solution. Worked with all major cloud OEMs as ODM for 30 years in manufacturing IT/devops/processes, mostly all project delivered on time, on quality, just by being flexible, communication, planning and good documentation making changes were reasonable. Getting high praises from the customers. Until young folks came in who did not have experience, discipline but were put in charge as account managers by other department heads , had to spend time to clean up their PITA to save our conpany and customer's a . . Top managers started to make decisions without informing the team. That broke the bill. Then I retired, and started do my own thing.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
