back to article How can you possibly test modern software fully?

The common assumption about software testing is that "more is better", and testing all the possible states and variable combinations guarantees you will find all the bugs. In the real world, however, there is not enough time or enough testers to test every combination of every variable. Not all bugs will be found, making …


This topic is closed for new posts.
  1. Godwin Stewart

    5 x 8 x 3 = 140?

    Before talking about complex stuff like software validation, maybe you should start with your multiplication tables?

  2. Anonymous Coward
    Anonymous Coward

    Complex Testing requires a calculator?

    Umm ... have I missed something? when did 5x8x3=140 !! ...

  3. David Norfolk


    Now corrected (I hope). You obviously can't rely on inspection for QA...

  4. Chris Miller

    Fixed, but ...

    ... only in 1 of 2 occurrences of "140". QA's tough isn't it :)

  5. Anonymous Coward
    Anonymous Coward

    Do not disengage brain

    "Great Idea!", I thought, "I could script all that and have my test plan produced in minutes"

    Then I looked again at line 2 of figure 24.

    Access | *Linux* | Web | IE

    How can I test that combination?

    (Yes I appreciate your example was a simplified illustration and this serves as a warning to check for more subtle issues when doing it for real)

  6. amanfromMars Silver badge

    Back scratching......

    "Access | *Linux* | Web | IE"....... Obviously that would be a job for an Open Source Tester/Programmer working in a proprietary Closed System/Windows Core.

    A sort of covert Secrets Testing for Mutual Benefits thing.

  7. Webster Phreaky

    Don't Ask Apple .. they NEVER test their software....

    ... they leave it up to their customers to "test" and debug Apple POS buggy software and OS X. Here's just a sample today! -

    QuickTime 7.2 breaking Rosetta CFM apps -

    iTunes 7.3.1 (#3): Problems syncing with iPhone -

    iPhone batteries don't appear to fully charge - (Oops that's shitty Apple hardware)

    Special Report: Troubleshooting Mac OS X 10.4.10 -

    And these A Holes in Stupertino California have the balls to run those bash Microsoft Ads! Bwah ha ha ha ha hah!

  8. Anonymous Coward
    Anonymous Coward

    Why do this manually?

    Why do all this manually when there are tools, both open source and proprietary, that will do all the work for you and produce scripts to boot? I won't mention them here, of course, but those interested can do a search on "pairwise testing" and you'll find tools galore.

  9. Robert

    version 0.1

    Don't forget the invaluability of beta testing. All the matrices in the world can't take into account unique user configurations.

  10. Anonymous Coward
    Anonymous Coward


    Starts to look easier than testing. see etc etc

  11. volsano

    Assumes unitary products

    The article assumes that (for example), Internet Explorer is a single product (with one set of expected behaviours) across any platform it is deployed on.

    Sadly, that is not the case.

    IE for Apple differs in many ways from that under Windows. IE5, IE5.5, IE6, IE7 are quite different products, as are their point variants. Also their behaviour alters according to the presence (and value of) a DOCTYPE header.

    All that adds numerous columns for what the article presents as a single entity

  12. Iain Cartledge


    The article only uses IE as part of the example, if it was to use all versions of IE it would clog the diagrams. The constant of note is that the number of tests is reduced. If you followed the same route with different versions of IE you would get the same conclusion, the path is irrelevant and only there to illustrate the point.

    140 = 5 * 8 * 3 + (5 + 8 + 3) + (1 + 1 + 1) + (1)

    Brackets for clarity. 5 + 8 + 3 for when one or two other settings are either outside bounds or unused, 1 + 1 + 1 for the settings outside bounds or unused (1 for each out of bounds setting) and + 1 for when all settings are outside bounds or unused. Unlikely in this situation, but I think that's how 140 was found.

  13. F Seiler


    Interesting, the pairwise thing is certainly something to keep in mind. 24 tables to show the concept is maybe a bit excessive.

    However, combinations are one thing - how about an effective approach at determining "dangerous" values in larger ranges where exhaustive tests are not even realistic for a single variable - like for floats and large integer ranges ? (Not counting through evaluation of border conditions like expressions evaluating to zero as effective)

  14. Peter Mellor

    Testing: the combinatorial explosion

    It is not news that the testing of software requires a large number of combinations of inputs (for black-box testing) or of combinations of paths (for white-box testing). "Exhaustive" testing (in the sense that all possible occurrences have been tried) is obviously impossible for anything other than the simplest programs.

    If you want a real example of a fault that only caused a failure under unusual conditions, I discovered that a failure occurred with my department's web-based diary system whenever I used it between 12:00 and 01:00 GMT to look at a week that lay within BST.

    Generally, I would recommend random testing (i.e., choosing test cases according to a realistic operational profile) in combination with a reliability growth model to recognise the stopping point, determined in advance as an acceptably low rate of finding new faults.

This topic is closed for new posts.