Ah, backups and the holidays. Back around the turn of the millennium, I was responsible for the care and feeding of an old AS/400 system. I was way in over my head, but the powers that be weren't going to hire a full time admin for that machine when users were moving from dumb terminals to PCs (running Windows 3.1) and backend was moving to Novell servers. My role with the big iron was more palliative care than administration.
Of course, the MRP software still lived on the AS/400, so it was still business critical, so when a hard drive died it was urgent to get the machine restored quickly. Two or three years in a row, we had failures right around new years eve, to the point that I joked that the machine and I had a standing date for NYE.
We had a service contract with IBM, so when a drive died it was a simple matter of opening a ticket and coordinating a time for the service engineer to be on site. I'd usually pull all the covers and vacuum out the system while it was down and we were waiting for the drive to arrive.
Swapping the hardware was pretty quick. For you youngins, a hard drive on this system was a 3U 19" rackmount box that held a whopping 400 MB. Our system had a rack full of those (I think we also had 2 different capacities).
Once we physically got the hard drive swapped, it was restore time. I'd do the ritual trek to the fire safe (stored in an adjacent building) to retrieve the tapes. IPL (Initial program load, IBM speak for "boot from tape") was done from a set of 4 1/2" tapes. Each one ran for about an hour, so I could catch a quick nap in my chair before swapping them. The data was saved to an 8mm tape, so much less swapping was needed.
I think our record was about a 24 hour turnaround from the time a drive failed until I had the machine back up and running again.
Biting the hand that feeds IT
