...and the band played on.
This is from a Rock and Roll Lighting perspective rather than IT, however we do use lots of IT kit these days. This happened one summer about 5 years ago, I had done a UK Stadium tour with a big Alternative Rock band, and during overnight programming sessions I had noticed all the APC UPS's dropping back to Battery power every time the Lighting Designer hit a big look - Lots of strobes and audience blinders - and I realized the sensitivity settings of the UPS's were too high for working on generators. I got the warehouse to send out a USB to serial interface and a serial cable and armed with the DOS based update program I went round all the equipment racks and changed the sensitivity to Low without having to turn anything off or disturb the Lighting Designers programming time.
A year later I was looking after the racks understage a large UK festival (big Egyptian shaped stage). I had noticed when loading in that there was one UPS that was too sensitive and had obviously not been on the tour the year before when I had reset all the others. Again I requested the necessary kit from the warehouse but it did not arrive until the last day of the festival. Undeterred, I had asked for the kit, so I was going to use it - and in my opinion ALL the UPS's the company owned should be set to Low Sensitivity by default as we often run on generators.
I go round the back of the rack, plug in the serial cable and fire up the software. No UPS detected, I try again, same result.
While I'm sat there pondering why this isn't working - I had no problems the year before - another member of crew comes running into underworld shouting that "ALL THE LIGHTS ARE OUT".
Panic Time - Slash is onstage above my head - there are 60,000 people on the other side of a black drape from me and they are looking at a band playing in a dim stage (thank god it wasn't properly dark)
I run round the front of the wall of mains distribution racks and the whole data rack I was working on is off, no reassuring blinky lights to be seen. I turn the UPS back on and a minute later everything is working again.
It was unfortunate that the Intercomm Masterstation which powers the headsets to talk to the LD at Front of House (and has big flashing strobes and sirens to attract attention) was in this data rack. It had taken theOperator several minutes to get hold of somebody at stage over a radio, amid all the noise, to come and find me.
It transpires that the serial cable needed is a 'special' one - with non-standard pinout - with the APC logo embossed on the connector. This time the warehouse had sent out a generic serial cable, liuterally the moment I plugged this cable into the back of the UPS the UPS had turned off - no beeps, no warning just off and I could not tell from the back.
I sent the serial cable back to the warehouse cut into lots of short pieces.
Biting the hand that feeds IT
