To err is human. To really tmux things up requires an engineer A reminder to check and check again in today's Who, Me? as a Register reader learns the true meaning of persistence. The latest confession comes from an exotic South American city and one of the country's many ISPs. Our reader, helpfully Regomised as "Paolo", was an engineer working for the company and, as is the case for so …
Yeah, the fun of "oh, I'll put that into the correct production system / the database / the version control / whatever LATER". It will bite you. Hard. (yes. yes it did bite me. hard).
I hope Paolo did spring for a couple of "adult beverages" after the team had stayed longer. In German glider clubs you have to buy a case of beer (and some non-alcoholic stuff) for several occasions, like birthdays, passing certain milestones (forst solo flight, licenses etc.), and there's also the acronym "WGU" = wegen groben Unfugs (due to fsking up - badly and possibly in a stupid way). This one would qualify.
(same thing in certain teams I worked in - you f' up, you stay, you buy beer for the rest who stays and cleans up your mess, I think it's a good policy).
For the love of god and sanity. Save in what you type. even better is logging the terminal session. Email it to yourself and one other.
Bin there, done that.
It's right up with "just take a backup of the running config before I touch something, and keep it in several safe plces". Murphy knows when there is only one backup and screws with that. And do the same when finished because I know someone is going to "improve" things a day or two later.
Once upon a time I wrote a small shell script. It takes a filename and makes a copy of that with todays date as a suffix (date+time if date exists: if I ever run it twice in the same second it will complain), or a given suffix ('cpdate file before_I_messed_with_whatever'). Creates messy directories, but that mess has saved my bacon a few times. Yes, keeping everything in git would be an alternative, but...
If only every manager who screwed things up had the same attitude of rewarding people who had to work extra to clean up the mess. I often recall the awards for working 'above and beyond' the call of duty, and no mention at all of the mangers whose scrimping, breach of working practices, and doing things 'on the cheap' had caused the issue in the first place.
As the poet had it :
"Theirs not to reason why,
Theirs but to do and die."*
*'The Charge of the Light Brigade' by Alfred Lord Tennyson, https://www.poetryfoundation.org/poems/45319/the-charge-of-the-light-brigade
I used to work somewhere where the networking was outsourced and the switches needed to be renewed. Turns out that the networking company didn't know what had been plugged into what and their guy unplugged all the cables before someone realised his mistake...
One of our guys then spent the rest of his weekend on his hands and knees crawling around the office noting down the number of each network port that was in use...
At Korev, re: cables.
At a previous employer whom shall remaine nameless to preserve my mental blocks defending my dwindling sanity, they had the IT team cabling up a new building for 1K+ workstations to several hundred bits of rack gear. We had to record the cubical port the cable went into, the tag number of the cable itself, and then the port on the patch panel of whichever bit of rack kit into which it plugged. At first we grumbled about this as a massive PITA.
A few months later the building itself had a fire in the between-floor-infrastructure that slagged quite a large amount of said cabling. We were suddenly VERY glad for the record we'd kept of what went where as it allowed us to figure out which of the goopy blobs of insulation were ones to repair & in what order.
In any project bigger than a mere home office, I've never begrudged the labeling of the cables, recording of cable#+port assignments, because it can save you days/weeks/months of crawling around on hands & knees trying to figure out which cables are live & which need to be replaced.
*Hands you a pint*
Drink up, it's thirsty work down there in the spaces beneath the spaces. =-J
When we moved house I had a happy week dropping cables from the loft into various wall cavities behind the plasterboard (most of the time spent probing to find whether there even was a cavity, bleeping cheap building practices mutter mutter). Felt very proud that I'd put coloured bands of tape evey few metres as each stretch was unrolled from the reel of dull grey. Even remembered to put bits of paper with the colour code into the little window on the wall panel above each socket. All very neat and tidy (well, except for the squid tentacles reaching into the back of the Ikea cupboard containing the patch panels).
Now, two spectacle prescriptions later, really wish that I'd written the codes onto HUGE labels and sod the teeny, tiny little window on the wall panels. Especially as, over the years, furniture and shelves slowly grew in front of the sockets in every room, leaving them in permanent shadow!
First day in my first job in a bank my new boss demonstrated how easy it was to remotely manage multiple systems from his desktop. He proceeded to shut down the main NIS server for the trade floor systems instead of the dev box he had intended, queue much screaming as hundreds of clients very slowly switched over to the secondary
This is why I only use tmux to make a session survive disconnection, never to run several sessions in one terminal. Not enough visible cues about which session I'm actually looking at.
I use many terminal windows, so that I know that one there is connected to this service. Over there is doing something else. Each of them may be running tmux if I think the session is important enough to protect.
I might still do something that dozy, but it's more difficult.
I use multiple sessions but never to different servers. If I want a different server connection, I'll put that in its own terminal window (well, OSX Terminal tab ).
They say that the reason experienced people are paid so well is because they've already got most of their screwups out of the way on a previous employer.
Have an upvote for using screen.
Same here, multiple screen sessions
In theory each machine has it's own xterm in which i have a screen session running.
However many a times the temptation to just ssh into the next machine in stead of spawning yet another xterm is too big.
Hence my shell prompts also include the machine name, current working directory and effective user
(or depending on what I do some other critical bits of information, like whether i'm looking at release or debug builds or )
I have coloured bash prompts for all of my systems (well... classes of systems - my laptop is normal linux-y, some servers are nasty garish cyan-like, some are more red). I have not yet f'd up (after that change).
I have this problem not only with tmux screens, but rather with the (sometimes stupid) number of open terminal windows...
I think I've heard of at least one person who had their terminal set up so that when they switched which server they're logged into, the xterm they were using changed the colour of its menu bar.
Using ANSI colour escapes in PS1 to have different colour prompts on each machine (both as ordinary user and as root) is definitely something people have done. Probably saved a few jobs by now. :)
My shell prompts always contain the hostname, are bright red if running as root and contain (SCRIPT) if scripting a command after the time I managed to fill a disk partition(*) after forgetting script was running.
(*) This was back when disk sizes were measured in MB. Doing it these days would be impressively forgetful.
Ah, I think I need something explained. I have some UNIX shell experience and almost no Linux, but I use one Linux system, by ssh, that drops me when I'm not looking. I'm usually not doing anything I can't pick up, but I've been typing
while true; do echo Robert Carnegie; date; sleep 60; done
- to keep my session going, but being confident that there's a more elegant alternative approach...?
I was doing some security testing for a client, was back at the turn of the century. They wanted their eCommerce system checked. I did some looking at the source code and marked all the locations, where they hadn't sanitized input and handed in my report and started testing the weak points.
The dev team and management didn't want to know. Running a SQL injection to display a list of users and their credit card numbers wasn't convincing enough... So I went nuclear on them, my next test included a SQL injection of "DROP DATABASE;--" Bye, bye dev environment...
The handy part about dropping the database is that you get immediate feedback from all the devs and ops currently working on the system, about whether the test worked or not...
Yes, but that isn't such a pain for the devs, the problem is quickly resolved and they can continue.
Having to re-load the database structure and restore the test data (thereby also testing their backup and recovery processes while we are at it) means that it is a painful learning experience for them, and maybe, just maybe, they will take that into account the next time around...
With my programming hat on, apart from quick and dirty internal routines, such as a source code calculator I wrote back in the early 90s, I've always cleansed the input as best I can, when writing the original code. It is much quicker and easier to build it in at the beginning than to come back later and try and find all of the affected pieces of code and make sure they are fully sanitised.
But, there again, I grew up with properly defined specifications, including error conditions and a full test-suite that already existed to test the code, so there was little wriggle room to not properly sanitise the inputs, otherwise it never got out of unit testing. When SQL Injection came along as a common attack, it was new to me, but we were already escaping most of the characters that would allow it to be successful.
On the other hand, on newer projects I worked on, the code was just written, basic testing to see if it worked with valid inputs and then it was thrown online. There were no test suites, nobody tested other people's code, you tested your own and most of the programmers hadn't learnt proper testing during their studies/training, so they weren't looking for errors occurring because the code was getting invalid data - especially SQL Inject type attacks, buffer overruns etc.
Yep. Whenever I'm writing code, I spend the first half of the time making sure that it can't do anything. I mean anything that it shouldn't do, by implementing input and output sanitisation, etc. I then spend the next half outputting lots of debug info when run with a debug flag set (which is often worth its weight in gold), and only then do I spend the final half of the time making sure that it does what it actually should be doing. And every segment of this multidimensional cake has an at least medium thick layer of comment icing on top, because there really is nothing worse than having to come back to something several months later, and then find out that it was rather more write-only code than you had intended…
But now everyone who has a name like José, or O'Neill, or the much-missed Steve Bong's personal assistant (etc), or who lives in a flat with a number/address like 12/3 Sandstone Avenue will be rightly searching for you with the sharpened pitchforks.
Sometimes a regexp is the right way to sanitise input, but there are also many times where a regexp can just cause unhelpful limitations on inputs that are required to be acceptable, and, for SQL queries, it is often better to use prepared statements or stored procedures or similar as a better way to work around the problem.
In my environment the sanitation is only needed for account names, AD-groupnames, passwords or similar things which sometimes can be unicode, but banning unicode makes administration easier.
When it comes to the displayed usernames or town names: I don't care, that is always unicode and handled as singular UTF8 object and not even remotely string-evaluated and always encapsulated. Even when a " or ' or & etc are in there -> no difference. This is not bash or cmd, this is powershell or something similar which can unicode right out of the box and does not stumble across weird characters, no matter whether Chinese, Korean, Russian, Klingon, Emoticon and so on.
Me: Why do you have the corporate security policies on the external firewall server?*
Client Security Admin: IT's ok, someone tried to get into it once and they couldn't.
Me: (Speechless)
But this was the same Sec Admin who genuinely had a BCP with a special page for if a room was unavailable in the building due to a Police murder investigation (and 'lack of staff availability' was NOT one of his expected consequences**).
* Yup, they sort of had a DMZ architecture for access to all things external, but their location of services and applications was weird. Oh and they had on average two level 1 incidents, where most staff could not do their work and there was no workaround every week.
**Think about it, murder - staff availability, hm?
I recall a conversation with a Novell Netware admin at a reasonably large company about why he needed a more secure SNMP setup, rather than just the defaults. Think this was on NetWare 3.12
He didn't buy that there was a problem, so I suggested that I could shut down his server, following a bit debating, we settled on me dismounting one of his new and unused volumes.
5 second later after all the how the heck did you do that and how do we stop it happening, there was a lot more discussion about SNMP and security.
Its amazing what a little demo does to help people understand the real risks.
Ah, Windows snmp. Its amazing how upset Windows Admins get when you turn their interfaces off when they have their screen locked....can't connect to the DC to authenticate to unlock the machine... After this they changed their standard snmp configuration so this wasn't possible
I sat in the tail end of a post mortem in India which was trying to find out why a production system had been shut down.
The site "expert" (arrogant, would not be told) was defending his corner. An external expert said "6 months ago we did a review of your systems and one of the top 3 recommendations was to enable the screen colouring; red text for production; Yellow for test. Why wasn't this done? It is a one line change"
The site expert tried to bluster his way out blaming someone else.
A humble (but skilled person) said "excuse me, we did create a change record, but it was never implemented....".
We had a comfort break, while management found out the status of the change record...... all the recommended improvements were waiting for the site expert to approve!.
We didn't see the expert again... but someone else, a team leader (rather than an expert) took over and within a week all of the recommendations were in progress, and calm and confidence spread throughout the team.
On Windows Server I set the background to display a bitmap of the server name tiled all over the screen.
So you can always tell which server you are logged in using remote desktop and mRemoteNG.
And, yes, I did shut the wrong server down. Once.
Others did the same, as it is not easy to discern between local and remote servers, and that was before I did the background bitmap thing.
Beyond the general concepts, none of this story means much to me.
But The general principles as applied to life do. Having several similar looking systems in the same place make a mis-selection almost inevitable.
A good and simple example. The Honda HR-V has the hazard and start buttons close to each other, near the centre of the console.. Though different shapes they are both reddish and about the same size. Inevitably drivers ( and yes, but not just me) will wish to thank a driver who's stopped to let us out of a side turning or parking slot or some such and switched off the engine instead. Which could be anything from embarrassing to bloody dangerous. The new e:H-RV has a few refinements, among which is that the Start button is now back to the other side of the wheel, roughly where you'd expect an ignition key to go.
The Windows filer has 'Delete' and 'Rename' right next to each other, so so easily mis-selected.
Edit: I had to fire up the A5000 and check why this has never been a problem with RISC OS. In the filer Delete and Rename are next to each other, but for Rename you go into another submenu, so you would never be accidently clicking on the main menu entry unless you were actively targetting the Delete option. In Windows both Delete and Rename have to be selected in the same menu, making a few pixels mis-aiming disasterous.
Well, at least it's not the brain-dead idea that old macs had, where you ejected a disk by dragging it to the "trash" (and no button next to the floppy disk drive to eject manually). Whoever came up with that UI idiom needs their kneecaps kinetically rearranging with a 12-bore.
Ah yes. The easy way to crash an old Mac. Insert floppy disk delete some files, then move the disk to the trash to eject it. Insert another floppy disk and do the same. Repeat ad nauseam. Then try to empty the trash bin. It required all floppies to be in the drive at the same time! So much for intuitive approach.
(Yes, I know I'm a hypocrite as I sit here typing away on my iMac, but this is a true story and was my first experience of Apple products.)
Ever since W95 we have had Maximise and Close buttons next to each other. That has carried over to become the default in KDE. It's not such a problem now because it's normal to throw up a dialog if close has been hit with unsaved data. Back then it was unlikely that any legacy application, a.e. just about everything, would make such a check because accidentally hitting close was so unlikely.
For those unfamiliar with GUIs pre-95 close was on the system menu, on the button at the opposite end of the title bar to the minimise & maximise buttons.
Tape jukebox I encountered in the late 90s (maybe HP) had a very bad grey-on-lighter-grey LCD screen with four or five rows of "button" labels in two columns, then the physical buttons to each side of the screen.
Unfortunately, I managed to wipe half the backups one day when I mistakenly selected "Initialise" (top right) instead of "Inventory" (top left). My eyesight is not the best, and in the dim server room with the very dim screen, one label with 8+ characters beginning with "In" looks much like another, and it was only the second time I'd exported the tapes from that device.
But I did know that "Inventory" took a matter of seconds to scan the entire jukebox (only 24 tapes). It unfortunately took me many more seconds to realise that hearing the tapes get loaded one by one and a few seconds of "bzzzzzt" to write the initial sectors was a bit unusual and taking way too long. At least when I yanked the powers supply when the light finally dawned, I timed it sufficiently well so that no tape got jammed in the heads.
roughly where you'd expect an ignition key to go.
Golly , imagine taking multiple tries to come up with that breakthrough.
Now we just have to wait for car manufacturers to figure out that keys that broadcast codes 24/7 that open your car for any Tom Dick or Harry do not really fall under the definitions of "key" or "security"
One flash of the hazards is a universally understood symbol for "ta" used when the person who you are trying to thank is behind you.
You're not one of those idiots who thinks that cars are metal boxes with wheels on the bottom that are used to transport people around, are you?
No, its not.
One flash left then right turn might be, more people just put up a hand as a wave of thanks.
I've only seen it as people approach queues of traffic - although the brake lights are sufficient!
My pet hate is unnecessary hazards on a parked vehicle, particularly when the view from the road is obstructed so all you can see is the offside indicators suggesting they might be about to pull out.
I've seen all of these. Just because there are multiple ways that have come into use for people to be polite does not mean those who use a different way are idiots.
Regarding the last point about the car looking like it might pull out - I guess the objective of the hazard lights letting you know it's there, has still been achieved.
Having just bought a bmw, what are these “indicators” of which you speak?
Joking aside, for years I’ve assumed people driving them just abandoned them wherever they felt like, but the amount of times I’ve got out, looked at how I’ve parked and went “shit, that’s fscking awful” and then got back in, I’m convinced there’s some reality distortion going on with the mirrors on them. I can park a 15M tri-axle bus easier than my car!
I've only seen it as people approach queues of traffic - although the brake lights are sufficient!
Not sure if it's still the case but I believe it used to be mandatory in Spain to use your hazard lights if you were the last car in a queue where a queue wouldn't normally be expected.
Doing a lot of motorway driving, activating hazards as you slow gently to join the back of a queue seems quite a good way of warning vehicles behind, particularly so if the queue is actually at a standstill and you have stopped and applied the handbrake. The number of times I've seen people in my mirror skittering all over the place as they suddenly jam on the brakes making the anti-lock work overtime is scary - especially so when more alert drivers have already seen the problem up ahead and prepared to stop.
M.
It’s also pretty standard on the M25.
A couple of years ago I had to emergency brake on the M25, as had everyone else in front of me, (uphill, anti-clock, approaching J4).
The hazards came on automatically. Which is just as well as everyone looked like they were shitting themselves.
"One flash of the hazards is a universally understood symbol for "ta" used when the person who you are trying to thank is behind you."
No, it's a universal sign of a bad driver. Hazards do not mean that. People who use them like that shouldn't be allowed to drive - it's a sure sign they can't do so safely or sensibly.
I was trained to give a very light tap of the brakes (enough to light the brake lights, but not enough to actually affect the speed) to give a signal to the person behind you that they might be driving dangerously close to you, just as flashing your lights at oncoming drivers is used to indicate that, for instance, they are driving at night with their lights off, they have a flat tyre, or there is something hanging off their vehicle, etc.
Hazard warning lights are to indicate that there is a hazard that other drivers need to be aware of (such as an obstruction in the road, sudden queuing traffic on the motorway, etc.), not to indicate to another driver that you think they are a hazard, and a single flash of the hazards after being let out by another driver is pretty much a universal "thank you" signal in any country I've ever driven in.
The one that almost caught me out was when driving in southern Italy (Naples down to Sorrento). When pulling onto a road - flashing headlights do not mean "after you" as they do in the UK but "keep out of my way I'm coming through".
The other thing I noticed was the speed limit and "no-overtaking" signs were obviously only for tourists.......
One experienced boatman, who had 'driven' motorboats on canals for years, on his first driving lesson in a car, was told to pull away from the kerb. He started the engine, put the car in first gear turned the wheel and moved away from the kerb, then turned the wheel the other way 'to bring the stern out' and crashed into the car parked ahead of them. Possibly the shortest driving lesson ever.
The scariest driving I've ever experienced was a taxi driver in Naples taking us from the airport into the city (we later found out that the bus is a much better option, for various reasons). Hurtling round blind bends in tunnels at about 50-60 mph, I dread to think what might have happened if someone had broken down in there.
You are a brave, brave individual to drive in that part of Italy, although I suspect it's a little better as you get towards Sorrento. I'm fine driving over dangerous mountain roads in Greece, where there's a sheer drop and no barrier, but I don't think I'd like to drive in Italy at all...
Try Lisbon, about 30 years ago, in a taxi with shot rear suspension cornering on damp cobbles: more of a slide than a drive. The added irony was that I had recently signed a life insurance policy whose wording had asked me to vouch, for the selected premium level, that I did not indulge in dangerous activities/hobbies such as parachuting etc. After Lisbon, I felt that their list was a bit short on real life experiences, but did not disabuse them of their ignorance.
Drove a car down south via Germany ending in Italy, Brother-I-L driving mostly. I'm kipping in the back in Germany, wake up to see us going through roadworks with an 80 limit, car doing 120... scream to slow the F down, he gets the hint.
Enter Italy from Austria, speed limit 30 for about 50km due to apparent non-existent road works (or to let the visitors enjoy the scenery). Cue us doing 30 as per instructions, km long queue behind us of locals all tooting their horns....
I was trained to give a very light tap of the brakes (enough to light the brake lights, but not enough to actually affect the speed) to give a signal to the person behind you that they might be driving dangerously close to you
I did that on the M25 years ago, when the clown behind me had closed up almost to to bumper-touching distance and was flashing his lights to get past, despite the inside lane being full of slow HGVs that I was passing and the outside lane moving slightly faster.
One flash of red from me, and he braked hard, greatly increased the gap, and only then did he think to turn on his blue lights... If he'd done that in the first place he'd have made much better progress
Tapping the brakes with a tail gator on your ass is illegal here.
BRAKE CHECKING is an illegal action. It is when a driver ahead of you deliberately and unnecessarily brakes hard in order to force you to take evasive action or to cause you to run into the back of them. ... It generally happens in a road rage or aggressive driving situation and is not an approved driving maneuver.
Reckless Endangerment I believe it's called.
I agree.
I don't like being hounded, but if you are too stupid to respect the security distance, it's your fault if I have to brake suddenly.
I won't brake for no reason, I don't want my car damaged, but I'm not going to pay attention to you if I have no choice. You'll hash it out with the police.
You quotation does not back up your assertion that what the OP described is illegal in your jurisdiction.
"BRAKE CHECKING is an illegal action. It is when a driver ahead of you deliberately and unnecessarily brakes *hard*" (my emphasis).
The OP referred to a very light tap on the brakes - enough to flash the brake lights, not enough to slow the care. That cannot be described as "braking hard".
UK Highway Code on hazard lights (all left and right turn indicators blinking):
"These may be used when your vehicle is stationary, to warn that it is temporarily obstructing traffic.
Never use them as an excuse for dangerous or illegal parking.
You MUST NOT use hazard warning lights while driving or being towed unless you are on a motorway or unrestricted dual carriageway and you need to warn drivers behind you of a hazard or obstruction ahead. Only use them for long enough to ensure that your warning has been observed."
I'm having trouble decoding that... is the scenario, someone comes up behind you in lane, you blink at them to say stay behind? And when they do, they have to blink at the next car behind them?
Otherwise: you are the hazard.
They are really expensive to use anyway - I know because most of the Scottish drivers I see stopped at junctions don't turn on their turn light until they start moving. (I am in Scotland.) An automatic power saver setting maybe? As a cyclist, it's similar, I have to stick my arm out and it gets sore, so I do drop it. But I re-signal before moving.
"Honda HR-V has the hazard and start buttons close to each other,"
That's up there with the Chrysler Pacifica. It has three round dials with bumpy edges within about 7" of each other (about 16 cm if you measure how many liters of fuel it holds).
One dial controls the fan speed for the cabin HVAC. One controls the radio volume. One is the gear selection control.
Yep, levers are boring, and Chrysler, formerly part of the Mercedes family, decided to bring over a "luxury" feature, useless as it may be.
I just drove to my father's house in a hire car (I no longer own a vehicle)*, and was provided with an Audi. The steering wheel is nice and round and turns. The 'station selection' wheel on the car stereo (sorry, 'vehicle entertainment system') is nice and round and turns.
So I'm trying to change channels, while driving along the motorway, steering straight with one hand on the wheel, looking at the screen to select a channel for the radio while turning the wheel to change the selection with my other hand. Not easy.
Rather like standing on one leg while rubbing your tummy and patting your head with your eyes closed. Fortunately there were no vehicles in the adjacent lanes as I veered all over the place, eventually decided that OK, I'll listen to what it is set to right now.
*Trying to save the environment, but honestly, having to learn how to use a different radio each time I hire a car, and all the paperwork I have to provide to prove I've not had any convictions since last month is beginning a to wear a bit thin.
Someone I worked with years ago was showing a new member of staff the as400 system menu. I will say I don’t know anything about those machine as it isn’t my area.
He was showing off the above command which tells the machine to shut down. There is a default timer which will wait for I think 30 minutes before running the command, instead of exiting the screen he hit enter.
Followed by a big warning this machine is going down in 30 minutes - there were about 1000 active users on the system.
Cue panic as they tried to get everyone off asap. The machine then took around 40 minutes to restart…..
"The machine then took around 40 minutes to restart….."
Whoa. Even worse than an old XP box I had the misfortune of using. I think it was a Celeron clocking around 450MHz, and 128MB of memory and a harddisc that was big, but seemed permanently stuck in PIO mode.
A perfect storm (imagine the swapping!) that made the machine take forever to boot, to load stuff, in fact doing anything was comically slow. It's when I first understood that error sounds were handed off to a different service than the error display. So I'd get some sort of dialogue, read it, select an option... And a few seconds later there'd be a "ding!" or "clang!". The first few times I'd wonder what the hell was going wrong now, until I'd sussed that...no...the machine really was that awfully slow.
No, not IP address, "Insufficient Paranoia".
When working with any sort of remote connection, you always need to double and triple-check the window you're issuing commands to.
And if you're not using that remote connection right now, for $deity's sake, disconnect for the sake of your own sanity.
you always need to double and triple-check the window you're issuing commands to
Indeed, you must. And even so I still managed once to fuck up. Somehow, it might have been a lazy Friday afternoon, I still mistook the machine's name after checking twice or trice. After the supposedly shut down machine kept responding ("huh, how did the command not get through?!"), a pair of sweaty palms and a restart everything was back to normal withouth anyone else noticing. But it taught me that I cannot be trusted with such machinery.
So far, I've not managed to make some of the mistakes that I've seen others make first-hand. I'd probably put it down to the fact that I don't need to do much sysadmin type work, and hardly ever need to go onto client's live machines. Thankfully, where I work, those roles are suitable ring-fenced, so it's hard for me to actually screw up in that way.
Having said that, I'm just waiting for the moment when I accidentally delete the wrong Azure VM.
I once (well, perhaps more than once) restarted a process on the production box when I should have restarted it on the dev box.
I put my hand up to the mistake (fortunately, there were hot standby systems and no-one noticed) and suggested ever-so-nicely that perhaps the systems should be modified so that people like me were not ABLE to restart processes on production boxes....?
I was working at my Sun workstation once when it just 'died'. I went to our SysAdmin of the time:
"Iain, my machine has just died."
"It shouldn't have."
"Well, it has."
Pause. Iain peruses his terminal and looks at the last commend he typed while rlogin'ed as root to my machine.
"Oh, sorry, my fault."
Instead of typing "Kill -1 1", he'd typed "Kill 1".
Safety scripts are a wonderful thing here. Years ago when I played with computers for fun instead of for a living, I got tired of accidentally turning off the server (in my garage, with the cold concrete floor) when I was working on it from my laptop from my warm bedroom or office or couch. Of course, this being an ancient server, there was no BMC I could connect to to turn it back on.
So, I wrote a very small safety script named 'poweroff' and stuck it somewhere in the path with higher priority than the regular 'poweroff' command. It simply checked if it was being called locally or across an SSH session; if it was local, it called the regular 'poweroff' command and so was completely transparent. If it was called from a remote connection, it would pop up a confirmation message giving the name of the machine, and "Are you sure?" A quick 'Y' would proceed to shut down the machine, while anything else would bail out of the script without doing anything.
Saved me more than a few walks across a cold concrete floor, and IMO something similar ought to be an option on pretty much every OS out of the box.
ICL's VME operating system would tell you if you had entered a command it did not recognise. Imagine my surprise on typing "Fred" followed by <return> and getting the cursor rather than an error message. I never did find out what 'Fred' did, but I don't think it was having a cup of tea...
(https://www.youtube.com/watch?v=Ge_4SlJWfl0 )
I have a couple of machines I don't need or want to be powered off* & to come back up automatically after a power outage**.
*These machines only have "two" options Disconnect (If RDP is used) or Restart in the shutdown menu.
** BIOS is set to return to previous state if an unexpected shut down occurs.
I hosed myself at work more than once. 8 boxes hooked up to a physical KVM, some running DOS, some running WIndows, some running Linux. Gave the proper logout and shutdown commands (pre-ACPI), flipped off the box's power switch, and ... D'oh! I'd typed the shutdown command into a different computer. Unique screen backgrounds and BGINFO solved that.
Once upon a time in the 2000s, the operators team got bored and since they got a nice training in VCS we'd just deployed for all critical services, they decided to make the setup "better".
Basically, one day, they decided to change all VCS configs, wrongly believing it would magically propagate to the OS: file systems mount points, ressources, directories, etc ...
We ended up with a completely misaligned config between the cluster config and OS objects. And we never were informed, of course.
Since those systems were more or less never rebooted, every single maintenance, planned or not, was a complete nightmare for the next year: upon restart, none of the services would come back online and 2 hours of trouble shooting and rebuilding of the config was necessary, each time.
As for the operators and their manager ? They nicely got away with it, never were acknowledged to have been the root cause by management, due to politics.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
