Non-shouty client call.
I had just returned from the wilds of Scotland and a major gas terminal there on a Friday night. Saturday morning I received a call from work to call the client, which I did. He very quietly said that there was a serious problem with the control system and the alarms and printer were going crazy and I needed to get back there pronto. Questions about what exactly was happening were met with further "you need to get back up here now"s and the Client's voice was very quiet and I could tell he was s****** himself, er, was very worried. About something, what I knew not.
No remote access in those days, so I climbed back on a plane (sheesh) and got there Saturday afternoon. Shure enuff, printer going bonkers, alarms having to be silenced every few seconds (it was a new system they weren't familiar with). I took a look and it all seemed to be just one alarm. Shurely they could see that? I asked what were they doing outside, on the plant. They had recently re-started flow from the offshore gas line and suddenly, alarms like crazy. Er, might these be connected? As Process Engineering was actually my original discipline, I could see that there were slugs coming in from offshore, and the alarm in question was the Slug Catcher liquid level low-low. The plant had been below that level and as the slugs came in, they temporarily raised the level then levelled out below low trip again, setting off the alarm, quite correctly. I asked how long did it take to restart the line, to which they said several days in all. The slug catcher was a monster beast and this was definitely going to continue for a while.
Just one of those things you learn that drop between the cracks of the functional spec. We can't know which loop(s) need filter blocks unless you tell us which loops are liable to jump around, we do try to ask you, but you can't see that such things are necessary if you don't realise what's going to happen. I configured a filter block and it all went quiet. Fortunately the flight south was not followed by another northwards until some good time later.