If only it were games
Back in the days of the Solaris 32-bit to 64-bit transition, I had horribly tight deadlines, as head of the distributed systems team at a regional oil and gas explorer, due to a very big seismic data analysis project.
I had a datacentre recently filled with Sun servers with Fujitsu SCSI RAID storage arrays with rebadged host SCSI cards that were randomly crashing, corrupting filesystems, trashing files in hard to identify ways, and all this in the days before ZFS. The vendors were of course finger pointing at each other and the clock was ticking.
We realised that the thing that sometimes, but reliably sometimes, crashed the boxes was ... doing an Oracle install. So I automated the Jumpstart for about 40 of the systems to do a clean OS install. Then do the Oracle install, and then report success, and start the process again. The boxes that failed didn't and a watchdog script emailed me as well as CCing my old Uni email account over the newly installed ISDN E1 internet link. If it was after hours, I'd run 5km to the office (I was fitter then!). I hand-noted the console errors into a journal.
About 4 weeks into this regime, I'd been working 28 days straight, and discovered the joys of working on Saturdays and Sundays - no distractions! - but getting well and truly sick of the process. Analysis of the console crash text was getting somewhere, the Sun (esp?) drivers were the probable cause, now we were into Monte Carlo testing of config options and continuing to complain to Sun and Fujitsu account managers every work day.
Building security noticed that I was in constantly, and barred me. The board negotiated a deferred beginning to my enforced home-time of another week.
Finally on the same day that Sun advised us that they'd identified a regression in the 64-bit ethernet (?maybe?) drivers, that we could fix by configuring them to run in 32-bit mode, we found a memory mapping in the bottom 2Gb of memory that gave us non-crashing boxes.
I wish I'd spent that time playing games.
Biting the hand that feeds IT
