back to article Mathematical approaches to managing defects

Software testing is still a controversial subject – everybody agrees that it is a "good thing", but it is frequently the first bit of the process to get cut when deadlines bite. After all, those sneaky testers are really responsible for the bugs, aren't they? Our software is just fine until strangers start poking around …


This topic is closed for new posts.
  1. QA Helper

    Consumer Software Real world

    Interesting article and from a QA perspective, from the consumer software world, way ahead of the game. Maybe this article if better suited for the financial software, device driver, embedded software, or aviation realms.

    As for consumer software ...

    Non-Ambiguous spec.? This in itself would be a revolution in itself and one which is really not defensible in many environments. The reply is that it would be a waste of money to write and to keep up to date. This is because by the time it's done, competitors would have released their software and the marketing window missed.

    Remember that your customers are not used to a quality product, and competitors are 'just shipping' as fast as they can. Still want to write a non-ambiguous spec (and get the kind of people on board who can)?

    Of course there are concrete things organizations can do to improve their QA / Testing (there's a difference, but one should at least walk before running), far before getting to using proofs and baysian analysis.

    These are all the old standbys and everything that worked before the dot-com bubble got everyone dreaming about hacking a quick bit of code and making millions from a buggy little prototype.

    Most of the things you can do are really in the development area. Ideally, developers should be catching everything they can, given the mentality needed to code software well.

    Quick examples, there should be ways to reward developers who do excellent testing, as well as those good at prototyping new features. Buggy frameworks should be avoided (think for a moment of how hard that really is!).

    In the testing area, you ideally should have independence from the developer mindset and should be able to focus on problems which developers can't imagine but which consumers can run into. It is normally a different person, and a different type of person.

    Back to the real world again, where QA staff get builds which sometimes can't launch successfully. Where QA is under an engineering manager. Where there is substantial ship pressure and quality expectations are not high.

    At that point, the much-maligned integration, even black-box, tests are the only ones which will save you. One has to test actual user / software interactions at some point, from the typical to somewhat rare. The product has to be tested in an integrated fashion. In the end, it's the only way to ensure the customer won't find it first.

    Isn't this inefficient and inelegantly blind and old fashioned? Subjective likes or dislikes aside, it doesn't have to be inefficient or blind. The judicious use of quality automation tools and scripts (don't let that also become a source of bugs and spiraling time costs) can gradually bring costs down a long way. Remember that Moore's law is on our side here, as is code re-use. Ask yourself how many problems are really new and never seen before.

    Not to knock any of the ideas in this article, but the vast majority of software still exhibit simple bugs which consumers find with the greatest of ease. The majority of these can also be caught with integration tests which are cheap and fast, if well automated.

    Once this approach has become unproductive, it may be time to look into much more advanced ideas. Even then, don't stop running those automated regression tests!

  2. David Norfolk

    Ahead of the game?

    Well, probably CbyC and Bayesian analysis are still ahead of the game. There are certainly development shops where even test automation would be a radical innovation. And possibly, quality in consumer software isn't that important (except that the games industry has to take bugs seriously - and silly bugs in wordprocessors and spreadsheets can really spoil your day).

    But I think it's the "attitude" I'd complain about - not that of "QA Helper", who obviously understands the issues and has made a reasoned choice (not necessarily one that I'd agree with, but I could be wrong) as to what is appropriate in particular environments; but the attitude that says "software has bugs, always has, always will; so let's not worry about it".

    Software doesn't have to have bugs in it; so if it does, hopefully someone has done a proper risk assessment for defect removal vs. delivery; but perhaps someone just decided that the customer wouldn't mind; or that the customer prefers low cost to quality. Or perhaps the developers are oncompetant - or. more likely, untrained and/or or badly led.

    Unrealistic? Well, possibly, but I mostly remember people regretting not getting it right - and stuff delivered too quickly taking even longer to deliver in the end. I also remember Toyota's success with "Lean" and how competitive British Leyland was in comparison.

    Remember, (and I do mention cultural issues) that there is some evidence that removing defects with formal methods is CHEAPER than doing it by testing (although you can't replace testing entirely). But perhaps this is only in fields where quality matters - not testing anything is cheapest of all, if someone thinks that production failures don't cost anything.

  3. Anonymous Coward
    Anonymous Coward

    Nice article, shame it barely scratched the surface

    It's nice to see mathematical approaches being mentioned in the mainstream e-press at last. What a shame that the piece on formal methods concentrated on just one company's system and failed to touch on the many other techniques being used.

    SPARK may be great for embedded aerospace systems, but if you are developing a large commercial application, you probably want at least basic OO features, like inheritance and polymorphism. Enter Verified Design-by-Contract, ESC/Java, Perfect Developer and Microsoft's Spec# (yes, even Microsoft is looking at FM these days).

    Or maybe you want to prove that your software can't get locked up in states from which there is no way out. Try a model checking tool - choose from FDR, Spin, SMV, SLAM... (that last one is from Microsoft again - maybe they're on to something!).

    QA Helper argues that writing a formal specification is expensive and time-consuming. Sure it is - but the formal spec is shorter than the code, and quicker to write. The trick is to use the spec to greatly shorten the coding time. We generate most of our code directly from the specification.

    Lot more info can be found at the directory of formal methods maintained by Jonathan Bowen of London South Bank University -

  4. David Norfolk

    Formal methods

    Well, I've written about Formal Methods before. And yes, we only scratched the surface - but a) I thought that treating one company in reasonable (for some value of reasonable) depth was better than the usual whirlwind tour of companies most people haven't heard of, with a bare para on each one; and b) this article was already about as long as could work online.

    But I'm happy to cover more formal approaches and in more detail, if those involved can put their case forward as well as, say, Praxis can (I didn't find Praxis through press channels, I was originally impressed by hearing Rod Chapman speaking at a BCS meeting). And, of course, if enough Reg Dev readers express an interest...

    BTW, I do realise that Microsoft has now discovered formal methods (I did mention it) but I'm waiting for the weekly bugfix stream to dry up before I get too enthusiastic.

    And I'll check out that URL - thanks.

This topic is closed for new posts.