Re: Need to check that failures happen when they should
I have a very real example of this.
We needed to reliably generate a callback anytime a 3D camera changed position or orientation (there were of course context and other requirements that complexified this).
The solution was very simple; after every "step" of the display thread, we would check if the transform for the camera had changed and generate a callback if it had. Overall this took maybe an hour to implement.
The randomized test took a week to write. A few of the complexifying factors:
1. Most user functions that could change the camera were processed asynchronously.
2. Some operations which were nominally setting the camera wouldn't actually change the matrix (for example, setting it to the same position twice) and therefore were not expected to generate a callback.
3. The test couldn't check if that had happened, because requesting information about the camera would force the test thread to synchronize with the display thread, defeating the purpose of the test.
4. If you wrote to the camera twice in a row, those two operations might get merged. We didn't guarantee that writing to the camera twice would generate two callbacks.
5. We DID guarantee that the LAST callback would reflect the final state of the camera, so of course that had to be verified in the test