Pay attention, class: Today you’ll learn the wrong way to turn things off Welcome once more to Who, Me? It’s The Register’s Monday column in which we celebrate your SNAFUS and rejoice in your recoveries. This week, meet a reader we’ll Regomize as “Steven”. “One of my first jobs in the 1980s was as a training instructor at the headquarters of CAD CAM company Intergraph,” Steven wrote. At the time, …
Day 4 of a new job.
I was asked to move some equipment below the items was the ups for the comms room with an emergency power down switch just 1 or 2 mm proud of the case, not recessed or with guards.
My knee caught the switch and dropped the power to the comms room.
Fortunately there were other people in the room at the time, and it must have been forgiven as I was still working there 20 years later.
At one previous place of work some utter genius put the "Press to exit" button for the server room next to the "Emergency power off" button. They'd clearly seen the trunking spur for the kill the power button on the wall and thought "Well, we can just run the door system straight off there" and bolted another, completely identical, breakout box off the side of the existing one about 4 inches away. They were even the same colour.
The expected happened three or four times before someone finally sprung for one of those little plastic covers that came down over the top of emergency power breaker button where you had to lift it before you could press the thing.
Is that I never got to push the EPO button. I've probably had access to one at nearly a dozen different datacenters ranging from my university to a massive datacenter that housed both a major Canadian bank and provided outsourced services to several US Fortune 500 companies. Would have been eerie to suddenly hear silence. At least for a split second, until I heard "what the hell did you just do?" lol
At my first job, we actually tested the Big Red Button. It was a mainframe, running batch jobs, so no need to worry about users outside the room.
The fire alarm went off. Being located over a paint warehouse, we ALWAYS evacuated if that alarm went off. We checked very quickly that no jobs were running which could not be restarted, then hit the button on the way to the door.
Everything went quiet except for the whistle of hard drives which took AGES to spin down without their brakes being applied.
Yes, the jobs were restarted 15 minutes later and customers were none the wiser.
In my first IT job I needed a power cable for something I was working on, and rather than walk to the supply room I decided to just steal one from a PC in the imaging room. I went to a random, unlabeled Dell Optiplex on the imaging bench, presuming it was just a computer we were prepping for deployment sitting on the backburner, and yanked the power cable.
Apparently that random, unlabeled Dell Optiplex was the machine running two virtual machines for the archiving department, it worked out okay though since I was the junior tech of the hour who managed to "figure out" what had caused the server to spontaneously go down.
A colleague was lifting a floor tile next to one of our racks and somehow managed to hot the power button on the rack UPS with the corner of the tile. He couldn't have possibly hit it that accurately if he'd been trying to press it with the tile. It's not been quite 20 years yet but he'll still be told to be careful with the tiles any time he goes in there for years to come.
I probably was not the only one...
Remoted in to a server on a remote site. Was also working on a server locally.
Did a shutdown on the wrong window, was supposed to be the local server, not the remote server.
At least there was somebody on the remote site to power the server back on.
Now I doublecheck to make sure I do the correct operation on the correct server...
Nothing teaches like experience.
"shutdown" does a system shutdown (all processes go byebye).
After that, you get a choice:
No other options[1], just sit there humming quietly.
* after the shutdown halt entirely, these days aka power down, as we now all have PSUs that can be controlled by software[1]
* after the shutdown, do a reboot
And other utility options to say when to do the above, whether to warn other users etc.
[1] BUT as we do now all have clever PSUs, "also power off" has been made the default, which probably explains why people appear to be thinking that that is now the primary meaning of the command, instead of the days when you'd do a simple shutdown (i.e. halt) and then have the chance to decide "do I press reset or just hit the Big Red Switch and go home now?). Often the latter, as you realise you didn't do "sync; sync; sync" before the shutdown and you can't face the output of fsck right now...
Originally, reboot immediately rebooted the system. No graceful shutdown, no signals to processes, no parking disk heads. Like pushing a reset button. It was only ever supposed to be called by shutdown in normal operation, but could be called by the admin on special occasions - for example the famous sequence
sync
sync
sync
reboot
Which was supposed to be the way to do an emergency reboot (where shutdown was not working as something wouldn't stop) with minimal chance of file system corruption.
Later versions of reboot would call shutdown if in a normal runlevel. And reboot if called by shutdown in the right state. Eventually you get the situation you see today. But old sysadmins get kind of nervous of commands like sudo reboot as they might be on that classic system, or a distro specially crafted by the BOFH and it might do what reboot was really supposed to do....
Similar, was doing a major ERP upgrade from one database system to another over Christmas one year while the office was closd for a week.
I could either remote from home directly to my desktop workstation or to one of our network servers, and then from the server to my desktop workstation depending on which port I connected to on the firewall This particular network server was our DC and print server but also controlled the firewall authenication.
At the time I had a thing about Windows looking like traditional Windows (blue desktop background, light blue task bar, etc), so our Server 2008 desktop was exactly the same colours as my Windows 2000 desktop.
This one day I had connected via the server as I needed to monitor things on the server while runiing updates on the workstation. After a few hours I went to shut down the workstation, but as you can guess I accidentally shut down the server instead. As the server controlled the firewall credentials I couldn't even log directly into the workstation and attempt to restart the server (it was running under vSphere).
I had to get the boss to meet me at the office the next morning, with 6 inches of snow on the ground, to let me in to restart the server. I changed the server desktop colour after that.
The other godsend is [on windows servers] removing shut down and restart from the start menu via group policy.
As you then tend to go for the next easiest option of shutting servers down via "shutdown -i" then it forces specifying which server you want to shut down and thus utterly eliminates the problem of "oops, wrong server".
I was managing a VAX 6300 in the early 90s, but of course I needed the assistance of the night-operators (who were more familiar with the as/400) to run the nightly backups.
my predecessor had built a nice interactive menu system to manage the backups in the All-in-1 system, but one particular night-op couldn't be bothered going back to his terminal to select the (next) menu item to rewind and eject the tape, and found an "eject" button near the tape unit. It was actually the "Halt" switch that did a hard reset on the machine, and as the machine booted it would rewind and eject the tape.
After a while I noted the short uptimes (less than 12 hours) were after the particular operator was on the late shift, and hung around to talk to him and ask what gives with the reboots. He didn't know, he wasn't (knowingly) rebooting the system, so I asked him to show me what he did each night. And got to the part where he would press the magic button to eject the tape.
I found the key and locked the console panel, and told him he had to use the menu on his terminal from now on.
no more mysterious crashes after that.
Worked for a large garage, where one of my roles was to stroke and fondle to PDP-11/73 that ran the franchise. One day I'm fondling away when one of the departmental manglers came to talk to me. And he decided to 'jokingly' tap the front of the PDP as if he was turning it off ....... mangling to hit the power button dead centre without even looking. His face turned red and scared ..... just as quickly as the silence descended upon us.
I told him to go, and not tell anyone about it. I restarted the PDP, and told my mangler that it had glitched.
And this is the first time that another person has learned of that incident.
I did once remote connect to the wrong PC and restarted it - I should have been more careful but two machines, one in our company and one at a sister company, had very similar IDs and I just connected and restarted without double checking as I knew our user was away from their desk.
It was noticed by the Dutch IT department but there was no comeback beyond a, "Don't do it again!"
Happens to all of us.
The most (potentially) desatrous I know of was a techie doing remote software updates, and subsequent restarts, on the control pods of a drilling rig's subsea BOP (blowout preventer, arguably the most critical safety system - think Deepwater Horizon).
He wanted to do that on a rig between wells, with the BOP safely stored on deck, but mistyped the address and basically disabled this last line of defense for an hour or so. Several people were decidedly unhappy. A pysical lock-out, with two independent locks, was subsequently installed, preventing any further unauthorized remote access.
Prior to win11, I always had my local machine's task bar at the top of the screen. Simple way to know if I am local or on a remote machine. Win11 still won't let you move the task bar to the top and MS have even blocked the registry hacks to force it. Company policy is all win11 (for desktops and laptops) so I can't even pick a diiferent OS if I wanted to. MS response on the feedback hub (where there are many many people requesting this) "We'll be continuing to evolve Windows 11 and its features based on feedback like this, so thank you so much for taking the time to give us your feedback!" - that was over a year ago.....
May take time, but they did finally fix the teams "15 favourite" issue
All I know is it took ages and no real fanfare until I noticed I had now added more that 15 - then twigged it is a bookmark style thing
Also seen that taskbar on multiple screens is finally coming. Not sure if that is for 24h2 or for 25h2
Not that I envisage going to either as 24h2 kills all my machines ( run like they are in a 5400 spinner with 4gm ram) and 25h2 from what I've read is just a few features on top of 24h2 :(
Yeah, I don't get that. IIRC they actually did a bunch of UI research back in the day to determine the placement and alignment of the taskbar/start menu and determined that the bottom left corner was the best place for the most commonly used button since it was the easiest to hit with the least accuracy. Simple ergonomic stuff like that seems to have been totally thrown out in order to appease some random designers latest need to for change for change sake.
Oh it's not change for changes sake. It's changes to keep their job sake.
If they went around saying, "you know what, everything is good where it is. People like it, it does the job, it's fine!", then Microsoft would happily say "Thank you for your contribution, here's your pink slip, dont let the door hit you in the ass on the way out..."
So in order to justify their emplyoment, (and pad their Portfolio for when they do get shown the door after all the hate arrives), they need to change things.
So I guess what I'm saying is that perhaps we shouldnt hate on the UI designers themselves, but on the gormless manglers who hire them...
It's like Logo redesigns, no "design" firm is going to come in and say, "Na. That Logo is fine. It's internationally known and respected, changing it will cost a ton and lose you business by confusing potential customers. Leave it as is..".
I do a lot of work in a virtual desktop (more network isolation allows certain essential programs to be installed that don't meet our current cybersecurity requirements for going on the main machine). After the n'th time of opening a program from the VDI taskbar instead of my laptop's taskbar, I switched the VDI to have the taskbar buttons centred, so I have a visual indication that this taskbar is not the one I am looking for.
I use Retrobar. Works fine for me, my only suggestion is that it should sort new tasks next to the existing "matching" task. I.e. a new explorer right next to the existing explorer(s) in the task bar, and not at the end. But still better than Win11 taskbar.
And I use vertical, with a customized "black Win2k" theme. Finally 20+ tasks visible at once with their text again.
Oh and that "Vertical and top taskbar back!" feature request is there since the very fist insider builds which featured the new Startmenu, i.e. since 2021. Is probably still number 1 feedback item. So much for "listening on feedback".
thank you so much for taking the time to give us your feedback!"
The only feedback they listen to is from their support/help-desk jockeys. "Hey boss, this call took 3.5 seconds longer because the user had changed some UI setting to put the taskbar at the top so when I asked him to click bottom left, nothing happened until I realised what he'd done" - Result: Stop users making changes to any of the UI design and layout because that affects the MS bottom line.
For now, at least, they still allow you to make the task bar left-aligned so things are mostly where you expect them to be instead of centred where *everything* moves depending how many apps/windows are open.
Reminds me of my first attempt at installing Windows 3 (or was it 3,1?) from multiple floppy disks onto my work desktop PC. In this case it was the ON/OFF switch being next to the disk eject button which meant a couple of mid-install power-cycling reboots and having to start all over again. :(
Someone told me about a major outage caused by moving some disks. This was 30 years ago when disks were boxes 6 ft high and 3 ft square.
There was not quite enough space in the machine room for some new kit - it was a matter of inches too small.
No problem. Two of them put their backs against the problem disk, and having been on the lifting course, pushed with their legs.
It worked!
The problem was that one of the guys got his belt caught in the Emergency Power Off button (which you have to pull to activate).
As they stood up there was a sudden silence around them.
It took a good few hours to recover.
They had been proud of their system availability, and had charts on the wall. Their carelessness was on display for a year, until it rolled off.
At one computer center at which I worked in the 1980s, my boss1, the director of the center, told this one on himself.
By "computer center", of course, I mean a large-ish room of hulking mainframes in full Blinkenlights glory, clothes washer sized disk drives, and dozens of twitching and whirling tape drives, along with associated fridge-sized controller boxes, etc.
One day he was giving a tour for some collection of big wigs and as the group passed the Big Red Switch in the midst of the tape drives, someone asked "What does that do?"
My boss, without thinking (obviously), calmly reached out, said something like "It's the emergency stop switch," flipped up the plexiglas2 cover, and pushed it.
Dead silence followed by near fatal embarrassment followed by utter panic ensued.
________________
1 Who I shall not name, since he is a nice guy and probably the best boss for whom I ever had the good fortune and pure dumb luck to work,.
2 I think Brits call it perspex.
Somehow DEC never got the memo that the power switch could use a little protection to prevent accidental or impulsive shutdowns. One team of testers I had the misfortune to encounter, would push that white button on a microVAX after any error at all, resulting in a lot of locked files whose status they helpfully added to the report of the original problem. They also had a habit of extending bug reports after any incident that they, in their ignorance, felt was related to the original.
So any fix that went in often was rejected because the bug report contained extra stuff added while the fix was in progress.
Of course, the large internationally famous company they worked for is now a pale shadow of its former self.
Oh, a very modern Cisco switch has a tiny on/off button on the front as well, an excellent place for idiots to run some cable bundles over.
I inherited this setup. One evening I'm adding a switch to the stack on request of our network guys, I accidentally move that bundle and didn't realise it now rested on that button. Until suddenly the fans of the stack went full tilt, thus announcing a reboot.
That tiny little innocent switch has a bonus feature: hold it in long enough and it does a factory reset. Which is what the cable bundle did. Which meant the entire config had gone. Which is jolly good fun if your (very good) network people are in another country which you now can't reach because your WAN just went the way of the dodo.
That heralded an evening with a hotspot, Teams and PuTTY to get the basics up (as that stack also powered all the Wifi via PoE), so they could then do the rest of the setup properly.
Yes, the next day I fitted a bracket..
> Somehow DEC never got the memo that the power switch could use a little protection to prevent accidental or impulsive shutdowns.
Actually, I have had a microVax - don't remember the exact type - that had the on/off switch and the power sockets in a locked compartment at the back of the case.
It was impossible to even unplug the power cables without unlocking it. At least, on that end of the cables. ;-)
But most other Vaxes I have seen just had a big on/off switch at the front without any protection.
Before I retired, my job involved quite a number of visits to North Sea oil and gas installations (inspecting and auditing various functions). I took the full induction tour each time (even if it wasn't mandatory, having been there in the previous year); I could have been on a dozen or so other platforms in between, so needed a refresher on where I now was. When the alarm sounds, you don't want to be running around trying to find your way to your muster point (and I've been offshore when it's sounded for real, as well as numerous drills). I had also made it a habit to try and forget each visit once I'd filed my report, to clear my mind ready for the next. Anyway, on one platform, the guide pointed out emergency shutdown (ESD) buttons in strategic locations; they had guards and everyone was told not to press one unless it was a real emergency (and, as newcomers, they shouldn't be alone out on plant). Of course, being told not to do something is a red rag to some people and, sure enough, somebody in the induction group asked something along the lines of "Why - what would happen?" and immediately pressed one.
After the furore had died down, everyone had been checked off and instructed to return to duty, and the platform production plant restarted, and the wells were flowing again (all of which took several hours and would have cost the company at least a million US$ production) the culprit discovered his first visit was also his last - he was on the next available helicopter back to the beach and barred from the platform. That may sound harsh, and he probably learnt a lesson, but deliberately doing something he'd been told not to do deemed him unsafe...
In the UK sector of the North Sea there's a contract convention that contractors (and, by inference, their personnel) are only liable for their own costs. Each contractor (and the operator) will (or should) insure for their own potential losses - not the losses of others. If that were not the case, every company would have to carry insurance that, for many, wouldn't be financially viable. In effect, your liability is limited to your quoted price for whatever work you have gone to do. Anything further, such as malicious behaviour, would be down to the courts to determine - and usually only enacted in the case of fatality or serious injury.
The opposite of that actually happened to me at a workplace long ago, not that far away.
The building guys had scheduled maintenance for one night, taking our UPS power (yeah, I know, building-wide UPS power, what a luxury!). So we prepared for that by shutting down all servers (DEC MIPS-based servers, yes, that long ago), syncing all disks and such at the end of the workday before that night, totally expecting that they would boot right up when power came back.
Only they didn't.
No UPS power was taken, therefore no boot. Which also meant we were greated by our servers sitting in the boot prom console (think BIOS setup screen) and happily blinking their cursor. And... nothing.
But since that day incidentally was the day my second daughter was born, and I just winked over from the nearby hospital (and yes, this was before mobile phones), I managed to get in the shortest workday ever, typing "b" and hitting the enter key on two consoles, at the same time telling my colleagues I wouldn't be in that day, and everything that the servers could manage to screw up while booting would be their problem now :)
Server room, remote support (Mumbai or Minnesota) wanted pictures of the layout & when provided requested physical button "S"* pressed, on-site correctly insisted that was the plant emergency shutdown, provided wiring schematics & stated there was no way in hell he was pressing it, without a clear mandate from the slaughterhouse manager to proceed once the remote guy emailed assurances that this was EXACTLY & MOST DEFINATELY the button he wanted pressed & that it would have no impact on our operations.
In between getting the confirmations as a email (Relevant Manglement cc'd in), authorizations, my colleague set about performing a full backup just before the timed operation & knowing what was about to occur timed it for one of the scheduled plant shutdown maintenance days (Not that they told Mr Remote that).
And so the time came around.... the remote guy was ready, backup performed, plant empty devoid of most workers except for the maintenance guys going in for a full clean up & replacement of bearings, rollers on the various conveyors etc.
What follows next is in the style of the Cornish Pub Joke, only as a Microsoft Support Call Agent advising that they have detected a problem with your computer....
Conference Call (For additional shits n giggles, they were using cellphones)
Are you ready to be proceeding with pushing the button.
Yes we are ready, are you sure this is what you want?
Yes, Yes I am sure, I am logged into the server & seeing all is well!
OK!
Please be pushing the button!
OK here we go!
The button was pushed to a sharp intake of breathe “There was a terribly ghastly silence. There was a terribly ghastly noise. There was a terribly ghastly silence.” Douglas Adams. as things powered down hard!
Goodness gracious what is happening there, I have lost connection to the server.....No err your plant, nothing is showing.
Well! What did you expect for hitting as we have repeatedly told you, provided schematics & pictures that it was the plant shutdown button!
I.. I... I.. will be having to report this to my management.
Please do, so will we & we have all the emailed assurances & confirmations you said it was safe to proceed & we will be billing your department\MSP for the downtime costs.
Errkkkk I will be getting back to you on that. (Funnily enough he never did, weird huh)
It took the best part of the day to get things back up & running (Or ten - twenty minutes after the first returning shift started at 6 am - That may have been the ILUVU virus that someone brought in on USB stick to print his daughters homework & took out worldwide operations) even after all the precautions.
*"S" may have been in his mind some UPS bypass switch or something he needed to work on - It wasn't.
"It took the best part of the day to get things back up & running”
Oh no, it should have taken you at least a few days to get things running. Let the company suffer, after all you have the logs to prove that you have done precisely what the obviously incompetent IT support told you to do.
Until senior management get hit with issues they won't learn!
Not made a power-off mistake, but my anecdote is from teaching IT skills in the late '80s.
One particular tiring day, I got fed up with a couple of students who for some unknown reason thought that the way to do a power reset was to flick the power switch AS FAST AS POSSIBLE. It had been a long day, and I stormed down the classroom and clouted the scrag with a (paperback) operating manual and shouted STOP TRYING TO DESTROY OUR COMPUTER EQUIPMENT!!!! There is *NO* resaon to flick the power switch as fast as possible giving the computer as little time as possible to realise what has happened other than to DESTROY THE FUCKING POWER SYSTEM!
Wow. That recollection felt nice, I need to calm down now.
Really, though. How on earth does it get into people's heads that the way to "turn off and on again" is to do it soooooooooo fast the power bearly flickers?
Every time we deal with large amounte of electric power we need to have a clear , in everyone's heads , well defined plan to shut things down and power them up. Whatever it is .. the PA or lighting of a concert system to large computer systems . Always discuss the plan before going into action. Everyone needs to know in as granular detail as possible the sequence of events to prevent damage to the equipment and personnel. Stay safe.
Yep, did the usual thought I was on UAT system and shutdown the company's main production trading platform at 10:45am on a busy weekday, this was a finnacial company that had $100bn AUM. I brought it back within 2 mins and blammed the network. The network team checked and by sheer dumb luck for me, they'd actually had a minor fault at 10:44am. They spent 2 days combing everything and in the end no one could work out how that network blip took the trading system off line.
I also once rebooted a company firewall from the outside, and yep I locked myself and the whole company out. I called my managed at 1am and told him, he was the nearest and had to drive 15 miles to the datacenter to reload the FW rules. We agreed it was a fault with the softwware, reported it as such, he claimed the expenses for the drive and got the rest of the day off and I was told to keep my gob shut!
Honestly this is exactly why I switched to using RetroBar on my Windows machines. The UI changes in Win11 that are “just for the sake of changing things” drive me insane, especially the taskbar being locked to the bottom and losing all the simple cues we used to rely on for knowing which system you’re on. RetroBar brings back the old-school taskbar look and actually makes it easy to tell local vs remote at a glance. It’s lightweight too, so if anyone’s struggling with Microsoft’s constant UI experiments, it’s worth trying.
