I had to do a close down and cold restart of an AS400 back in the early 90s
The machine was the main machine for the Granada organisation, you know, TV Broadcasting and production , Bowling centres, Nightclubs, TV rentals, all sorts of things.
I sent a memo out telling everyone I was going to come in on a Sunday Morning and do the shut down at 4.00am until about 6.00am so that people should not be using the system very much, but giving them lots of time to make sure they are definitely NOT using the system.
Now by this time the machine had been running for over 4 years and had never been turned off.
(It had had weekly tape backups done on Sunday mornings each taking several hours and done when there were very few people likely to be on).
So there I am, Nervously waiting until four am, shutting down all the sub-systems until it was all the way down.
Then turn the key to position 2, (if I remember correctly), press the red button, turn key to position one, and watch as the system shuts down.
It felt like I was euthenising a great beast.
So there I am, a wet behind the ears System Operator (and for reasons too complex to go into I was also the System Security Officer, which is not recommended), and I assisted the Engineer as we swapped out and upgraded various components and memory.
We were done quite quickly and by about 5.00am, just as the sun was rising, we sat back and lit a cigarette to celebrate (this was 1992, a long way away from smoking bans and sitting in a highly air cleansed computer room).
We decided to restart the entire system.
So, press and hold Green button.
Beep and cursor on the Monitor... Excellent.
Move key to position 2 and remove key.
I then start all the sub-systems one at a time, reset the batch files and set them all running.
We leave the building and go home satisfied with a job well done.
I had the Monday off because I had worked the Sunday.
Or rather... I should have, but didn't
8.00am Monday morning phone rings off the hook.
The whole system is up and running, but all the batch files are running slow, VERY SLOW.
2 hours for a simple purchase ledger invoice batch file slow.
Remeber this is a major company, a billion dollar, world wide company.
This is not good.
An hour later I am at the cliff face going through the sub-systems making sure they are all running, which they are.
I am testing memory, it is fine.
I am looking to see if all users are correctly assigned and that the access protocols are correcct.
Three hours of my life and nothing looks out of place.
At this point I decide to do the one thing no-one ever does.
I open the manual and look to see what I should have done when starting the computer.
Ahhhhhh!
When I restarted the machine the machine automatically starts a system called MASTER, which is a stripped down system that allows you to run a basic functionality routine.
When you start all the sub-systems you are supposed to turn off MASTER.
What had happened is that EVERY SINGLE BATCH file was being run through the subsystems, which completed their jobs very quickly but then passed the job onto the MASTER to rerun them.
Of course this small MASTER system only had limited memory and very little processing power, so was taking forever to do the batch jobs.
There were thousands of batch jobs queued and waiting, with more hitting every few seconds.
I switched off the MASTER sub-system and watched as the jobs started whizzing through the systems and clearing.
(It was still going to take quite a while to clear out the delayed batches but not too long)
I reported to the Technical Director, (who knew nothing about AS400s) and told him I had managed to fix it because I was brilliant and found the cause to be [Insert whatever BOFH/PFY generic techno babble] and he was very pleased.
I told him I would Tuesday off instead of today which he signed off on.
I went home and decided that a little day drinking would be pleasant on a Monday afternoon!
Biting the hand that feeds IT
