Re: Approaches to testing
We had a HA system which worked fine. Then one day, the primary failed and everything jumped over to the secondary. Which promptly failed and everything failed back to the primary (which was just about getting back on its feet) Which failed, etc. , etc..
It turned out one client got into a very weird state and sent command packets to the servers which the servers couldn't handle. Once we managed to get some logs and isolate the client the system became a lot more stable.
Biting the hand that feeds IT
