Causes of software development woes

"Agile development" can mean different things to different people. To some it's about easing up on traditional rigour, and even legitimising a quick-and-dirty approach to getting stuff out of the door. To others it's about implementing a different kind of rigour, in order to bust project backlogs in a more robust manner, and …

[Anonymous Coward]

And that's why...

You should always have these things in writing and signed off by someone at the highest level...

So that, if someone comes whining about wanting some feature changed or added, you can simply tell them that it's going to cost them extra...

[wolfetone]

Re: And that's why...

Lovely idea, something I've done myself. However, you don't take in to consideration the following issue:

The stakeholder - in my instances the guy who pays my wages - see's it as your job to do what he says. If he changes his mind, then it's up to you to do it again. The fact you've agreed to something is irrelevant to the fact that you're employed to do what they ask for, and if you've wasted your time that's not their concern.

To be a developer today is to be a navvy on the railways 100 years ago. Do what you're told, doesn't matter how tough the job is, if you don't like it then there'll always be someone else to replace you. Don't ever kid yourself in to thinking you're the Alpha and Omega of a project and can't be replaced.

[Anonymous Coward]

Re: And that's why...

and if you've wasted your time that's not their concern.

But I haven’t wasted a second of MY time. Only the time my employer is paying for anyway. If they want to send me off on a wild goose chase, that’s up to them, it’s their money.

[wolfetone]

Re: And that's why...

"But I haven’t wasted a second of MY time. Only the time my employer is paying for anyway. If they want to send me off on a wild goose chase, that’s up to them, it’s their money."

That all depends on the way you look at it.

If you're a 20 year old developer, fresh out of school, that is the sort of mindset they have. It's all about the money.

When you get older, your priorities change. Would you rather work longer effectively repeating your work for the sake of an extra few quid, or would you rather it was done properly and agreed so you could go home at a respectable time and put your child to bed?

There's no right answer to it, and I'd agree with both statements. However, right now, I would only agree with the latter. My 20 year old self though would've agreed with you.

Life is far too short to treat any gobshite boss's demands as wasting their money. It's wasting your time, and you only have a finite amount of it.

[Anonymous Coward]

Re: And that's why...

Would you rather work longer effectively repeating your work for the sake of an extra few quid, or would you rather it was done properly and agreed so you could go home at a respectable time and put your child to bed?

No, I don’t work longer. The way I see it is my employer buys from me 40 hours a week, and they may spend those hours however they see fit. If my boss misallocates my time then that is on them. I make sure there’s the evidence if I ever need it that a days work for a days pay has always been done.

[JohnFen]

Re: And that's why...

"The way I see it is my employer buys from me 40 hours a week"

That was the way I saw it at the front end of my career. Now, the way I see it is that my employer is paying me more for my expertise, experience, and insight than for the number of hours that I work.

[Andrew Moore]

Re: And that's why...

"No, I don’t work longer. The way I see it is my employer buys from me 40 hours a week, and they may spend those hours however they see fit. "

I think there is a vast difference between being employed as a programmer and being contracted to develop a specific application. In the first instance you are most likely salaried; in the second it's most likely to be a fixed, negotiated price. The problem with the second scenario is when the client decides that they want more than they paid for, and are refusing to renegotiate a price and/or are withholding money owed.

[Doctor Syntax]

Re: And that's why...

"would you rather it was done properly and agreed so you could go home at a respectable time and put your child to bed?...Life is far too short to treat any gobshite boss's demands as wasting their money. It's wasting your time, and you only have a finite amount of it."

Look on the time you spend after you've gone home as your time. What you do with it is funded by their money. If they're determined to get as little as possible for it that's their problem.

However, a good plan if the instructions are verbal is to follow up each change of mind statement of requirements with an email saying what you understand them to have said. That set of emails becomes part of the project documentation.

[John Smith 19]

"statement of requirements with an email saying what you understand them to have said. "

This is the # 1 way to CYA. But make sure you keep an off site, off line copy, to avoid "The email archive got deleted" defense.

One hopes eventually the smarter PHB's will cotton on that what they say has consequences and the more accurate they can describe what they think is wanted the better chance they have of getting it.

At least that's the more positive view of doing this.

[Doctor Syntax]

Re: "statement of requirements with an email saying what you understand them to have said. "

But make sure you keep an off site, off line copy, to avoid "The email archive got deleted" defense.

Goes without saying. Also, include it in the project documentation that gets distributed to all the project team. That way it becomes more difficult to deny that the off-site copy's kosher.

[Adam 1]

Re: And that's why...

> That set of emails becomes part of the project documentation.

I would caution about using emails as project documentation. In 5 years time when some intended is being called a bug, the email from <insert old broom from customer who was frog marched out two years ago> to <insert old manager who jumped before they were pushed six months back> is not going to help you if those mailboxes are long dead.

I would suggest using something like confluence to capture the requirements. If you think that the customer (which could be an internal customer) is likely to try something on then by all means attach an email as pdf as a sign off to wave around later, but don't rely on outlook as a knowledge repository. Please.

[Anonymous Coward]

Re: And that's why...

Oddly enough, I've just started adding emails of this nature to a 'documentation' directory in the main git repository. Anon because the need to do this shows, well, management not having their shit together really.

[Lysenko]

Re: And that's why...

Life is far too short to treat any gobshite boss's demands as wasting their money. It's wasting your time, and you only have a finite amount of it.

I look at it more in terms of being a criminal defence barrister. You're probably going to fail 90% of the time and the client will regularly undermine both you and themselves, but in the end, that's irrelevant. You don't argue the case because you expect to win, you do it because you enjoy the intellectual challenge of the process. If the client blows up the project then they're the one paying the fine, not you.

[Paper]

Re: And that's why...

Agree with wolfetone. Sometimes it's more than about money.

I enjoy completing a project, and I want to make sure that the results please the customer, and their customers. So if they want a minor change before release that will take me a couple of hours or less, why not? It feels good to produce quality (and it gets you noticed too).

If it's going to take longer, I would inform them of that but then suggest alternative shortcuts around that so that they can get their product out of the door.

[JohnFen]

Re: And that's why...

"When you get older, your priorities change. Would you rather work longer effectively repeating your work for the sake of an extra few quid, or would you rather it was done properly and agreed so you could go home at a respectable time and put your child to bed?"

This is true -- and that's why when you reach that point in your career, it becomes very important to not put up with the sorts of employers who are strict "my way or the highway" types. Fortunately, you probably also have enough experience that you can find a better employer elsewhere.

In my younger days, I was just happy to be working in my chosen field and would put up with quite a lot of BS for the opportunity. Now that decades have passed, I'm much pickier about who I'm working for, and have no problems quitting if the match isn't good.

[John Smith 19]

And that's why...

you need mechanisms in place to handle that.

Except most British firms I'm aware of are a bit s**t about capturing requirements, let alone tracking them, seeing how the system delivered conforms to them, what's outstanding.

Because that would usually (to PHB's) translate as "Spend money on non-core software that does not directly help the business."

And so the fail continues.

[HmmmYes]

Agile is not a methodology.

Its inexact way of trying to define what you're meant to be doing.

No problem with that, as long as it recognised that its a small team creating a prototype.

Might be differerent when you have 50+ people working on a release, to a deadline.

[Adam 1]

Agile is fine

The problem is when someone thinks that it equates to "unplanned", or that changing requirements has no consequences.

When you boil it down, the claims it makes are hardly controversial. "Issues discovered early on in a process are cheaper to remedy than those found later", so tasks are supposed to be self contained a and be achievable in half a day. Sprints are equally a week or two so idea to usable feature is much shorter. If it is found to be a dumb feature, it is not likely to be intricately linked to hundreds of other features and therefore ridiculously expensive to change.

Analysis and design is really a Goldilocks artform. Too little, and the inevitable feature request comes in that requires an entirely new implementation even though in the customer's eyes, the request is simple. Too much, and projects get stuck in analysis paralysis, too scared to head down a direction because of unknown unknowns, or worse where the requirements change and developers become unwilling to throw away that code they have so much mentally invested in, even if a clean slate would be a better spot to start.

Of course, it is the unit tests and continuous integration that makes it possible to deeply refactor code without risking breakage, and these are often what companies won't invest in. It is always seen as an overhead, and the future efficiencies it creates are never credited to it.

[Paul Herber]

Dear architect, please design me a new house, I'm not sure how many bedrooms I want, somewhere between 2 and 7 should do the trick, some on the ground floor, some upstairs, maybe. I'll also want a good sized basement for the snooker table. I don't think the house will need much in the way of foundations as the rock in this area is solid granite. I might also want an underground garage for two or three cars as well as a workshop and storage space. You can draw up plans for all the possible options so I can decide what I want. I'm happy to finalise the design there and then but bear in mind that my wife will have something to say on most aspects of the house. You should also consider discussing the design with the in-laws, they will be unhappy with whatever you design but will not say a word about it until the build is finished. Please take this into account.

[fandom]

What makes you think that architects aren't treated like that?

Actually, I have twice the misfortune of being neighbour to an architect remodeling his home. You would think that, being architects, they would have the knowledge and the experience to plan everyting in advance.

You would be wrong, it was more like coming one day to tell the workers what to do and coming back the following day to tell to tear it down and do something else.

[DropBear]

Ehhh, remodeling these days seems to be considered hardly a bigger deal than reshuffling furniture repeatedly until you like it, only you're reshuffling non-load-bearing walls instead (granted, hella unpleasant if you live next door). On the other hand I have doubts that construction of a new house would commence without an existing complete set of plans (such as they are), or that the architect would be willing to change anything past that point without discussing supplementary compensation up front...

[T. F. M. Reader]

@fandom: "What makes you think that architects aren't treated like that?"

Most peoplemanagers have an intuitive idea of the difficulties and limitation inherent in building houses, bridges, railroads, etc., even though they have never worked in the construction industry. They realize that a) laws of Nature pose some limitations on what is feasible, and b) once something has been built changing it will be very difficult and costly and disruptive. They may not have a detailed understanding of the issues inherent in building a 100-storey skyscraper, but they intuitively feel that a 2000-storey building may not be a feasible task for a 10-man crew. It is probably the same kind of intuition that made them decent football players in the 6th grade despite lousy marks in Newtonian mechanics.

Thus, your typical laypersonmanager will not tell an architect "I said I needed a house for a family of four, but actually the house may or may not be converted to a major hotel 6 months or less after construction is completed" or "the bridge must be 10 miles long, it should only have support on the right bank, none in the middle, and the left bank end must not touch the ground at all" or "don't worry about water and sewage pipes at this time, we can add them after the building is built and populated, let's see first if more than a few tenants ask for them".

However, if anyone tells me he has never encountered very similar statements relating to software requirements I will assume that he is very new to the business.

[albaleo]

I'm guessing architects get such requests all the time. They're perhaps a little luckier in that they can whip up a quick sketch to show some ideas, and hopefully they can produce an agreed design that forms the basis of their contract. And all this before a brick is laid. In my experience, we don't have that luxury with software development. We need to build some software first before we can test it out on some real data. And if you work in education like me, you can be pretty sure that the first real data you run it against looks nothing like the sample data you were provided with three months earlier.

[PatientOne]

"I'm guessing architects get such requests all the time."

Indeed, it is common, and not just for Architects. The main difference, however, is the earlier the changes, the easier to impliment, and late changes can be very, very, very expensive.

With Civils projects, and with building in general, adding to the side isn't that bad, but adding another level means working from foundations upwards. In IT terms, it means changing the core code, then checking each and every associated module.

That's not to say Construction can't be agile: They can work to the specification and deliver those parts that are completed as they're ready. It's simply that you can't keep changing the design as you go without major cost consequences. Just ask the Scottish Parliament about that one :p

[Bananimal]

What's to stop you using a UX professional to research, wireframe, prototype and test before you build it? You get early feedback from the users, you have better defined requirements before you start building.

The excuse that you need to build it first, or that you need real data is rarely true in software development where there is an interface exposed to the user. It can be faked for early feedback, issues can be ironed out at low cost prior to build.

[Adam 1]

Now try not to cry when you watch this.

[Crisp]

Can you do us a sort of dashboard thingy with graphs and pie charts and stuff?

Me: Sure, what kind of statistics do you want those graphs to display?

PHB: I don't know... Make something up.

8 hours later...

PHB: Oh no, that's not what I wanted at all. What do any of these numbers mean?

Me: Nothing, you told me to make them up.

[Anonymous Coward]

Re: Can you do us a sort of dashboard thingy with graphs and pie charts and stuff?

Yah, except when you utter your cute punch line "Nothing, you told me to make them up," you get torn off a strip for insolence and obstructionism - and it goes in your personnel file.

[Dan 55]

Re: Can you do us a sort of dashboard thingy with graphs and pie charts and stuff?

"This is a test GUI generated with agile methodology to iterate towards the desired result. At the moment these fields are placeholders, what user story would you like to implement?"

Same thing, sounds much better, like all of agile. And it doesn't get you ticked off.

[Doctor Syntax]

Re: Can you do us a sort of dashboard thingy with graphs and pie charts and stuff?

Yah, except when you utter your cute punch line "Nothing, you told me to make them up," you get torn off a strip for insolence and obstructionism

That's why you confirm your instructions by email. And you use less cute, more business-like terminology: "You suggested that arbitrary data be inserted as place-holders until real data is available.".

[JohnFen]

Re: Can you do us a sort of dashboard thingy with graphs and pie charts and stuff?

"and it goes in your personnel file"

I stopped caring about what goes in my "personnel file" decades ago, since the contents of that file don't mean anything outside of the company that holds the file. The only "permanent record" that impacts me is the one that affects my general employability, not the one that affects my standing at a particular company.

I only care about doing the best work that I can. And you know what? That goes in your "personnel file" too.

[Mark Honman]

Foundations

Using the building analogy, it is possible to vary the construction along the way, as long as the building's foundations are sufficient. You can even add a second story if the original foundations are deep enough.

Extensions are a pain because their foundations will have to be designed to become as one with the original foundations.

In terms of software requirements, then, it's important to get the big picture of what the system might grow into, as a starting point for system design. That allows one to make appropriate platform and system architecture decisions that should prevent the system running into a brick wall as it grows.

There are 2 problems, of course:

The more capable the platform & more sophisticated the architecture, the longer it takes to get to something that stakeholders can see "turning its wheels".

Ultimately every successful system hits that brick wall where its requirements have outgrown the foundations. Succession planning is essential; we should be thinking about scheduling "moves to a new building" well before they become necessary.

What remains very difficult is making the case for the cost-effectiveness of a re-write; that as a system's actual requirements diverged from the requirements on which its design conception is based, there is a crossover point where the cost of maintaining multiple levels of workaround becomes greater than the cost of a re-write.

[Anonymous Coward]

Re: Foundations

"Using the building analogy, it is possible to vary the construction along the way, as long as the building's foundations are sufficient. You can even add a second story if the original foundations are deep enough".

Indeed, this is a large part of my understanding of software architecture. It involves creating a design with built-in dimensions of flexibility - and those dimensions should correspond to the types of change or extension likely to be requested in future.

Of course, that's far from easy. It calls for a Jerry Weinberg level of insight and understanding - of the organization and how it works as well as the software and how it works.

[Doctor Syntax]

Re: Foundations

"In terms of software requirements, then, it's important to get the big picture of what the system might grow into, as a starting point for system design. That allows one to make appropriate platform and system architecture decisions that should prevent the system running into a brick wall as it grows."

I can't upvote this enough. Ask yourself what's the general problem of which the initial requirement is an example. Solve that general problem. It might not necessarily be much harder, if at all, than solving the initial one and it will stand a very good chance of being easier than the problem defined by the changes of mind between the start and the deadline.

[Anonymous Coward]

"it's important to get the big picture of what the system might grow into"

Beware, it could also lead to an over-engineering trap - if you don't have really clear "what the system might grow into", and how long it will take to get there.

I've seen system designed as if they needed to store and analyze the CERN LHC data, instead of a medium-size company. Often, one of the reason was to sell them as much stuff as possible, and earn from that.

"Solve that general problem." - another risky proposition. The "general problem" may require far more complexity and resources, and if you don't plan to solve the same problem over and over again, you may find you have to tackle so many dimensions a solution is very hard to implement, and when done, it doesn't still solve the "general problem". It's called the "framework" approach...

[Doctor Syntax]

Re: "it's important to get the big picture of what the system might grow into"

Beware, it could also lead to an over-engineering trap - if you don't have really clear "what the system might grow into", and how long it will take to get there.

But beware also the under-engineering trap. I had a client who had a slew of small applications, each hacked out to solve almost the same problem. Every time a customer gave them a slightly different job they wrote a new application to do it. Could a single application replace them all and tackle all the other jobs in the same category? Yes it could.

And they still didn't learn. The next, far more complex requirement came along. The contract required two types of job, both basically take in the data (in XML) rearrange it to drive the production process, batch it up, inspect the result and recycle items that failed QA into the next batch. Result: they insisted on writing two systems, one for each job.

The next, more complex contract after that I got far more of a hand in specifying as well as developing. Most of the code to be inherited from the previous one got thrown out (in retrospect I should have thrown more out) and the replacement made more versatile so, with minor extensions, it coped with more and more additions to that contract and a subsequent one. After all, when you looked at the general problem it was just a set of rules engines to provide different bits of the functionality - and being XML, XMLT is a pretty handy rules engine. Just throw in some new rules (mostly style sheets).

[Anonymous Coward]

"But beware also the under-engineering trap."

Sure - you have to find the sweet spot between the two extremes. Usually tech people have a tendency to over-engineer (but the truly lazy ones), while the customer and management to under-engineer to reduce costs or increase revenues. Good requirements and processes need to mediate, and pin-point the right solution taking into account the constraints, which could be also users and maintenance people skills, etc. etc.

[Zippy's Sausage Factory]

I rant and rave at people who advocate agile as being "agile is the way to do everything, you're either agile or you're doing it wrong, agile will solve all your problems", because generally they're trying to solve the wrong problem.

The problem is that "agile" is such a buzzword for management that, like most buzzwords, they don't see past it and believe it's a magic bullet. "Never mind the problems caused by the business, clearly the problem is the developers aren't 'agile'. So we don't have to address our actual problems, just tell IT to be 'agile' and it'll all be fantastic."

A lot of this problem is caused, or at least made worse, by the slew of terrible books about "how to implement agile" that spend most of their time parroting this ridiculous utopian notion that you don't need to do any actual thinking, or make any changes to the any other part of the business, just adopt "agile" software development processes and everything will come right (agile in this case being nothing to do with the agile manifesto, but whatever snake oil training said "agile practitioner" wants to sell you).

And yes, there are organisations who actually make agile work, but they are few and far between as far as I can tell, and I've tended to find that they would probably work pretty well with almost any project management process, for the simple reason that they encourage clear requirement finding and definition in the first place. Which is, let's face it, the thing that actually makes the most difference anyway, regardless of whether you adopt agile, PRINCE2, waterfall, SSADM or any other requirement management process.

[Doctor Syntax]

The problem is that "agile" is such a buzzword for management that, like most buzzwords, they don't see past it and believe it's a magic bullet.

And the fate of management buzzwords - achieved in a very short time - is to become meaningless.

[Anonymous Coward]

Been there, done that, ate the pie

'A lot of this problem is caused, or at least made worse, by the slew of terrible books about "how to implement agile" ...'

I'm right with you - because after spending years working with people who used the waterfall process in an extremely detailed and well-thought out way, I had the misfortune to get another job editing software books. Many of which were exactly along the lines you criticize.

More times than I like to remember I had to challenge assertions in the introduction that "no successful project has ever been accomplished using waterfall". Oh dear. Should I start with OS/360 or the various NASA packages that got people into orbit and onto the Moon? How about every avionics package that controlled any plane you flew in? Or perhaps Windows, or Linux, or the RT/embedded OS of your choice... and on and on and on.

[adrian lynch]

The user is not your enemy

[Dan 55]

He bloody well is.

[JohnFen]

What a strange stance to take, to consider the people you are working to properly serve and support -- the users (a/k/a the customer, a/k/a the people paying your wage) -- your enemy.

I wonder how common that attitude is, and if it's connected to the general decline of software quality over the last decade or so.

[Anonymous Coward]

@ adrian lynch

Indeed, usually it's your manager or the salesman who promised the impossible or the psost sales team who screwed up customer requirements etc ...

Usually problem somewhere between dev team and customer

AC, obv

[Anonymous Coward]

"The user is not your enemy"

The *real* user usually is not, unless you're in a situation when they are happy with the actual situation, and someone is forcing unwelcome changes from the top.

Just, usually, to get at them, you have to get through layers of people who actually don't use the software but are in charge of some part of its acquisition, so they need to have their say and look "smart".

One of the key action for good requirement is to identify the real users and get at them, look at what they need, what they expect, maybe suggest - subtly - ways to improve their work - don't force "revolutions" if it's not what they really ask for. Most managers, if users are happy, will accept a solution because that makes their job easier.

Yes, you can still meet the old jerk, you have to understand how to neutralize them, when possible.

It's a social skill - something you also need to gather requirements.

[Dagg]

The user is not your enemy

Correct the bloody BA is the enemy!

[Anonymous Coward]

Eliciting requirements is hard

One issue I often encountered, is that people tasked to eliciting requirements are not really the right one. In this situation there are several ways to do it wrong - you could be too business oriented with a lack of technical understanding, leading to impossible or very expensive solutions, you could be too technical, leading to superb technical solutions that don't address the customer needs, or be a useless office document filler.

What you need is someone able to understand the customer domain well enough, also with enough technical knowledge to "guide" the customer when needed towards a good solution, without forcing it into the wrong one just to sell or use one technology or the other. They need also to be able to communicate and manage requirements properly - especially requirements are not marketing brochures or letters to Santa Claus. People writing requirements MUST NOT come from sales or marketing.

These people are hard to find, and means you can't believe to be able to deliver any software project you may encounter.

Then you need proper tools to manage requirement lifecycle and their relationship with tests.

Unluckily, most of the tools I've used, even expensive commercial ones, are usually uncomfortable to use. Since they moved to web interfaces, even more so. We would need really "agile" - in the simplest and truest meaning of the word - tools - tools quick and comfortable to use, not tools that need more time to be cared for than actually creating the product.

I need something easy to navigate, manipulate, change, using several views and dimension at the same time, in different windows when needed, and everything kept in sync with local changes without roundtrips to a web server,l and remote changes when needed. Something that works as well with a mouse and without it, when your hands are both on the keyboard. I don't care if it doesn't have a phone app or a mobile-friendly site - I don't manage requirements on a phone, it's the worst tool.

[Doctor Syntax]

One tool I used to use was Enterprise Architect. In particular it had an option to sketch out a user interface and then add on actions in the manner of a Use Case diagram. You could them produce a narrative of what happens if the user undertakes a particular action. For instance a drop down menu could have actions associated with User selects "Missile alert" and User selects "Test missile alert". It gives the user an impression of what the interface will look like, what actions will be available and what will then happen. It also serves as a straightforward guide to implementation as it provides a list of event handlers and what they should do.

[Vanir]

Enterprise Architect CASE tool

@Doctor Syntax

One of the best CASE tools around for the price. Just don't say this in a programing job interview for a company that does 'Agile' - requirements? we use TDD for capturing requirements! Scrum companies employs user stories which is use cases to me.

'Agile' practices are the antithesis to an agile mindset.

[Dan 55]

Here's the UI for anyone who's interested. It's atrocious.