one specific sequence of events exposed a code defect in our DNS service
Wot, no comments?
The moral of the story is...
Look for bugs rather than them find you.
===
I used to work in a real-time environment where the mantra was "if it falls over once, try it again. If it still fails, investigate."
If I did a test which fell over, my mantra was "I may never get another opportunity to squish what may be a possible one-in-a-million bug, let me take it."