* Posts by QA Helper

1 publicly visible post • joined 14 Aug 2006

Mathematical approaches to managing defects

QA Helper

Consumer Software Real world

Interesting article and from a QA perspective, from the consumer software world, way ahead of the game. Maybe this article if better suited for the financial software, device driver, embedded software, or aviation realms.

As for consumer software ...

Non-Ambiguous spec.? This in itself would be a revolution in itself and one which is really not defensible in many environments. The reply is that it would be a waste of money to write and to keep up to date. This is because by the time it's done, competitors would have released their software and the marketing window missed.

Remember that your customers are not used to a quality product, and competitors are 'just shipping' as fast as they can. Still want to write a non-ambiguous spec (and get the kind of people on board who can)?

Of course there are concrete things organizations can do to improve their QA / Testing (there's a difference, but one should at least walk before running), far before getting to using proofs and baysian analysis.

These are all the old standbys and everything that worked before the dot-com bubble got everyone dreaming about hacking a quick bit of code and making millions from a buggy little prototype.

Most of the things you can do are really in the development area. Ideally, developers should be catching everything they can, given the mentality needed to code software well.

Quick examples, there should be ways to reward developers who do excellent testing, as well as those good at prototyping new features. Buggy frameworks should be avoided (think for a moment of how hard that really is!).

In the testing area, you ideally should have independence from the developer mindset and should be able to focus on problems which developers can't imagine but which consumers can run into. It is normally a different person, and a different type of person.

Back to the real world again, where QA staff get builds which sometimes can't launch successfully. Where QA is under an engineering manager. Where there is substantial ship pressure and quality expectations are not high.

At that point, the much-maligned integration, even black-box, tests are the only ones which will save you. One has to test actual user / software interactions at some point, from the typical to somewhat rare. The product has to be tested in an integrated fashion. In the end, it's the only way to ensure the customer won't find it first.

Isn't this inefficient and inelegantly blind and old fashioned? Subjective likes or dislikes aside, it doesn't have to be inefficient or blind. The judicious use of quality automation tools and scripts (don't let that also become a source of bugs and spiraling time costs) can gradually bring costs down a long way. Remember that Moore's law is on our side here, as is code re-use. Ask yourself how many problems are really new and never seen before.

Not to knock any of the ideas in this article, but the vast majority of software still exhibit simple bugs which consumers find with the greatest of ease. The majority of these can also be caught with integration tests which are cheap and fast, if well automated.

Once this approach has become unproductive, it may be time to look into much more advanced ideas. Even then, don't stop running those automated regression tests!