Red rag, meet bull: The software resilience gamble

You, the fine Reg readers, recently regaled us with the gory details of your application failures - and it ain't too pretty. It turns out that a large majority of you find business is disrupted by app failure way too often. Of the 1200+ readers who took part in the research, a whopping 84 per cent said their business suffered …

[Anonymous Coward]

operational IT feels largely ignored during the software development lifecycle.

that's because 90% of i.t. departments software engineers are tasked to work with refuse to have any input, as they are more than happy with the world they have set up - all nice and cosy, with no superiors really able to get a handle on what they do day to day. Then these smug engineery types breeze through, trying to make the world a better place (that's a world where the engineer gets paid, nowt else) and the i.t. gimps see it as a dominance display. Unable to compete against intelligence with their bteq in network admin, they retreat inwards, only occasionally popping their heads outta their asses during meetings to make some snarky f*cking comment about how if they'd been involved from the begining we'd have none of this trouble now. To which i can only answer, if ya mum had strangled you at birth with your placenta then we'd not be in this situation now, and the world would be a marginally nicer place.

[Keith T]

A former profession in deep trouble

The quickly decreasing status and slowly decreasing responsibilities of programmers is driving down the intellectual level of programming departments.

A lot of CIOs are salesmen. They sold themselves and got the job. But they mess-up their staffs because they don't know how to represent the IT technical side of things to upper management, they can't think on their feet concerning technical IT issues -- they don't have the expertise. It is similar with CIOs who are accountants. The executive committee is sitting around, and instead of IT expertise, they have an extra salesman or accountant.

In some companies this lack of expertise extends through middle management down to project leaders, project leaders promoted because they could not do the technical side of the work.

Finally, the intransigence of IT people, refusing to learn the business, refusing to bring forward their own ideas, refusing to take responsibility for technical decisions. We are the very definition of irresponsible.

[Unlimited]

Investment

^^ That's the key word. Most of the wobbly applications I've seen were implemented due to cost and time reasons.

Client wants application built cheaper, but won't compromise on features. Result: testing is reduced, or they go with a supplier who ridiculously under quotes, and may not even be capable of delivering the requirements.

Client wants application that does everything they want, but don't want to invest the time in fully specifying what they want, or in user testing.

Or, the most common scenario: The marketing manager has set a deadline without consulting anyone, and so development is rushed.

[Phil Miesle]

MTTF

More components = more likely any one component will fail. The concept is that simple, though the math is a bit more complex...

How, then, is it surprising that as businesses incorporate more and more software they have more and more software failures?

[John Latham]

Resiliance is a feature...

..and just like any other feature, it costs money.

People WANT bug-free, 100% available software but are not prepared to pay for it because they don't NEED it.

Your report mentions "business impact", but spending money has business impact too - you could do something else with the cash.

In most organisations, senior management understand these tradeoffs, and put up with griping from below when things break because it would too difficult to tell people "your occasional thumb-twiddling time is cheaper than the IT spend it would take to give us five nines availability".

In the real world, people buy German and French cars over Japanese because of superior look and feel, despite poorer reliability. Why should we expect choices to be different in software?

[stu]

When will Engineering principles ever reach IT

As a EEE in IT, and an engineer through and through (takes his car to bits, builds his own sheds, knows how to build stuff generally), I lay the blaim firmly at the feet of the 99% of 'computer scientists' whatever the feck that means... most people doing IT are not engineers. Even if they did 'computers' at uni, it will have been a comptuer science course 9 times out of 10.

I have never understood why building software is any different from building a bridge, a car, a house, a shed, or an amplifier. Yet the people we have building software wouldn't know basic engineering principles if they hit them on the head... to use an EE analogy, they would happily spend 2 months building a 1.234 ohm resistor if that is what the calculation demanded rather than using a 1.2 ohm off the shelf.

They have no concept of quality as a variable, an absolutely fundemental engineering principle in the real world and no less so in the virtual world of software.

And finally, they have this incredible pious attitude when encoutering a problem which always seems to lead them to 'how on earth can we solve this problem that has clearly only ever been seen by me and my company and never anyone else i n the known world. An engineer sees a problem, and expects the solution to be builld somewhere already most of the time, and knows to use one of them nearly every time.

muppets.

[James Condron]

@ Keith T

"The quickly decreasing status and slowly decreasing responsibilities of programmers is driving down the intellectual level of programming departments."

Not to mention the idiots our Universities and Schools are churning out; over heard on a programming class (java) on a software dev. degree:

"Okay, so I want to change some stuff... can't I just decompile and do that? Its taking me ages to rewrite my code"

Or that guy's friend, he wrote a class to test text for certain things, anagrams and palindromes, few other bits. He called it 'String'... whats more his String class was using java.lang.String, he spent three days on this code before he asked a lecturer, trying to work out why it wouldn't compile.

[Brett Weaver]

@john latham

Lack of resiliance in software is the result of employing people who are poor at their job. It costs less to write it right than it does to write it wrong.

Its just that people fail to understand that development skills are just that. Skills.

Bill Gates estimated the difference between a good and a poor developer as being 10,000 to 1 ...

Unfortunately, corporates, especially corporates with strong HR departments, cannot understand, let alone retain talent in any department.

Thomas J Watson Jr is credited with the quote:

"A man is judged by the company he keeps - A Company is judged by the Man it keeps"

[xjy]

Engineering principles = pie-in-the-sky

Stu's harking back to the old days of engineering glory when you had to be able to do a Frankenstein on stuff (rip it apart and fuse it together again) to use it and keep it usable. When the Chief Engineer was respected, and the Royal Engineers the backbone of the military.

Bloody utopian guff these days.

It's all "Give us an algorithm and bugger first principles" these days. And everything's all-in-one and inaccessible to users. No maintenance. No repairs. Everything done by magic. Hawking and bean-counting rule. Coalface experience means nothing cos coal is over and done with (or soon will be).

There's maybe one real engineer per conglomerate, who sketches out something that's put together and assembled by numbers. A few semi-engineers who can fiddle around with the assembly process. And lots of the rest of us who hang around waiting for some pretend engineer to strike lucky and plug our downtime hole with chuddy-gum. Like last year when after tooling around for a year trying to get my special donglized software running on our upgraded XP intranet network (it worked OK on the old NT one) - the network is BIG - they just gave up and gave me a whole new extra computer minus the intranet to run it on. The cost in labour time and labour wear-and-tear must have been astronomical.

Maybe I should just have done engineering like the rest of the family...

(Paris cos she knows all about hands-on experience...)

[Anonymous Coward]

JFDI

The most problems during application development are caused by a lack of forward-planning by management resulting in JFDI (Just Fucking Do It). They pick random figures out of the air and then use them as target dates / batch windows etc. Then its up to the developer to make it happen within this smaller-than-required development window. Against your better judgement, you're forced to cut corners during testing and in my place - heavily feature on El Reg - peformance testing is usually an after thought that gets dropped due to time contraints.

[Anonymous Coward]

@keith T

"refusing to take responsibility for technical decisions."

yep, pretty accurate, most developers will refuse to take any responsibility for making technical decisions*, due to it not being worth the hassle. Generally it's not even noticed if it's the correct decision, but has dire consequences if it isn't. There are just no benefits to making the right decisions but massive risks for making the wrong one.

*At least decisions that have visible outcomes to non-technical people, we are quite happy to make massive, possibly even contraversial decisions when it comes to internal designs and code.

[Anonymous Coward]

@stu

"they would happily spend 2 months building a 1.234 ohm resistor if that is what the calculation demanded rather than using a 1.2 ohm off the shelf."

I actually thought engineering was all about precision. I work on banking systems and i have a funny feeling most people wouldn't like it if we used interest calculation algorithms that were about right, instead of exactly right. Although i don't think most people would come away particularly happy after seeing our algorithms because of their precision. Remember the 5.5% the government has legally said banks must tell you(1 decimal place) can actually be anywhere between 5.45% and 5.54999*% and it makes a difference on compound interest calculations which one gets used :)

Using your car analogy i don't think i'd want to be driving one where the engineers had used off the shelf parts that were about the right size instead of exactly the right size.

Although you are right in that a lot of places don't need programmers, they need developers*, people who know engineering principles of reusability, maintainability etc are a must. and know that taking the time upfront to define & create exactly what is needed saves money in the long run against using something approximately right, but that causes things to go wrong.

A programmer can write code from a design, a developer can write the design too, it's a difference that a lot of university 'computer science' courses need to learn.

[R J Tysoe]

Why does it not surprise me...

... that the commentors can't even spell "resilience".

[Mister Cheese]

@R J Tysoe

Or "commenters" for that mater...

[yeah, right.]

Example

One sets out and implements a working solution to an IT problem. One applies standard engineering principles (Yes stu, not all "computer science" folks fail to understand them) to the design, implementation and support of said solution. One applies standard business principles (because those have been studied and applied as well) to issues surrounding the solution, including return on investment and ensuring that the solution resolves the business need it is supposed to solve.

In other words, one creates a solution that works.

One is then told to change the solution by some muppet in management who thinks he knows IT because he once wrote a spreadsheet. Protests that proposed solution will fail go unheard, because protests through the management chain are intercepted by lying, self-serving managers, and protests outside the management chain are simply ignored. Like most managers, this is one who got their job because they lie, cheat and steal with impunity and gets lots of practice because that's what management does. Unfortunately, you don't get much practice lying, cheating and stealing in IT, because computers can't be lied to, cheated or stolen from.

Solution breaks. IT guy gets blamed, not the fucking asshole manager who ordered the mess in the first place, or the illiterate butt-kissing sycophantic accountants or salespeople who usually populate the management layers of IT.

Lather, rinse, repeat. Which is why I work for myself now, so that I can simply ignore these lying scumbags if they decide to screw around with what works. Dilbert isn't just a cartoon, and the pointy-haired boss isn't an oddity, it's the standard.

So if there's an IT problem, don't blame the people working in IT. Look at the management, because that's invariably those who created the problem in the first place, and it is invariably that class of creature that perpetuates the problem.

But they're really good at deflecting blame.

[duncan wood]

@stu-AC

Every car ever built is made from bits that are "about the right size", it's called tolerancing, you do it to every single bit & then work out wheter or not it's likely that all the bits will still fit together. The Japanese dominated by working out which expensive bits didn't need close tolerances & which cheap bits did

[duncan wood]

Spelling resilience

A resilient spell checker might be usefull for addresses a well :-)

[Kevin Kitts]

GIGO...

The management wants the problem solved for a cost minimum. However, for a little more, you could create the optimal solution. Optimal solution is denied.

2-3 years later, it's time to upgrade. Poor solution requires twice as much work as the optimal solution to upgrade, but for a little more, it could be upgraded to the optimal solution. Optimal solution is denied.

3-4 years later, it's time to upgrade again. Poor solution requires 4x as much work as the optimal solution to upgrade, but for a little more, it could be upgraded to the optimal solution. Optimal solution is denied.

See the pattern here? Crap-code snowballs over time, until the point when it becomes totally unworkable, and the management throws up its hands and says "we need to spend a huge amount of time and money to get XYZ software and train our people, because we can't make our software work anymore".

If you do things right the first time, the code lasts a loooooooooooooong time. And you get what you pay for. Anything less, and you pay more for it over time.

[Dr Patrick J R Harkin]

@Anonymous coward

Awww, diddums, didda nasty net admin delete all your pr0n again?