After deleting a web server, I started checking what I typed before hitting 'Enter' It's Monday morning, and a week of possibilities presents itself to IT pros everywhere. Which is why The Register brings you another edition of Who, Me? It's the reader-contributed column in which we remind you what not to do with your day, your week, and your career, by sharing stories of your worst workplace mistakes and the …
Wilma was extremely lucky that there were backups of the other web sites, but her first mistake was to begin by DELETING the old web site directory, when a more experienced person would have RENAMED the directory e.g. "mv website website.old", before installing the new web site alongside it. That way, if the new version didn't actually work in the production environment, the old web site could be restored in seconds with a couple of directory renames.
Still, the wise learn by their mistakes.
Tee-hee.
MGR: GET RID OF ALL THESE "DOT OLD" FILES AT ONCE! Use version control like a professional!
Me: Can I have git installed please Mr Manager?
MGR: No. We have a windows version control tool.
Me: What's it called?
MGR: Dunno.
Me: Who administers it?
MGR: Dunno.
Me: Where can I find the documentation?
MGR: Dunno.
<months later>
MGR: What are you doing?
Me: Using my Stevie-built version control. It's a script. Before one edits, one types in the name of the script and it copies the old version to a "dot old" file, then you can work in safety.
MGR: Idiot! What about previous versions of the same script?
Me: Simps. Before I do the "dot old" copy I copy "dot old" to "dot old dot old"
MGR WHAT?
Me: Works a treat. when I need to get a previous version all I do is take the version number, subtract one and look for a script with that may "dot old"s appended.
MGR: Why did you do that? I TOLD you to use the windows thing!
Me : Yes, but you couldn't tell me what it was called, where it was to be found, who was in charge of it or where the documentation for it was. In the face of so many insurmountable obstacles I was forced to improvise. I was going to put a link on all our servers, but I realized that was hard so I just put a copy of the versioning script on every server we have.
(here the temptation to say "it's good isn't it sir? it's got three engines etc" was overpowering)
Of course I had done none of that (well, OK, I *had* done the "dot old" versioning script for giggles), but this chap had not only refused to have git installed, he was in charge of the SA twerps who had deployed all the trad Unix version control utilities for root access only.
From my Tae Kwon Do grandmaster:
"Strength is not equal to knowledge, and knowledge is not equal to training. But combine knowledge and training, and one will acquire strength."
I suppose you could substitute "Wisdom" for "Strength" here. However, I also appreciate the distinction D&D and Rolemaster provide for stats, when comparing Wisdom and Knowledge.
I grow varietals of cherry tomato that are sweet enough to put into a fruit salad. Tasty. Unexpected. Recommended.
Intelligence is knowing that a Watermelon is a fruit. Wisdom is knowing it is actually a cucumber, and works well when sautéed with chicken (or pork chops) and served in a chipolte tomato sauce.
Hint: Get a melon as red and sweet as you can find. Use a melon baller for the watermelon. Don't warn your diners. They will bite into one of the balls, expecting a cherry tomato. The look on their face is priceless ... as is the empty plate soon after.
"Only a fool learns from their mistakes. The wise learn from others' mistakes."
Let him/her who has never made a mistake, call those who make mistakes fools.
It can be salvaged with a few changes:
"Only a fool learns only from their mistakes. The wise also learn from others' mistakes."
Ah, but this was academia, where the computers are bought "just the size they need to be" and no bigger.
I well remember the humiliation of talking with a peer in a different company. She advised me to "simply rename the current application root directory and install fresh using the old name" and I had to sheepishly admit that my government workstation had 500 meg standard hard drives e'en though the World had moved to mutli gigabyte drives years before.
I've always used the convention of appending the date to backups, so I can have as many backups as I want, and I know their age. so the directory named webserver would back up as webserver-old-250915 (or webserver-old-250915.tar.gz if I archive them). Scripts in bash or powershell can easily do this. (And using the date in year-month-day order causes the names to sort properly, too.)
Fortunately she is in good company. The only reason any of us are able to give that advice is because *every single one of us* has committed this exact mistake at least once if not more.
She’s now off to commit some other fun like accidentally putting a space after the first slash in
’rm -rf /tmp/foo’
If you're deleting the old website to add the new, that means downtime, you have two directories "current_website" "new_website" and you update the host redirect to point to the new site, migrating with zero downtime...
Then you can do what you like with "current_website" when you've had chance to check "new_website" worked fine.
(and because I spend too much time on Reddit, before the pedantry, you wouldn't actually use "current_website" "new_website" as names...)
I don't rename, I make a zip
$ zip -r website-2025-09-15.zip website
I like using zip on Unix since its a format that windows users think they understand.
Works pretty much everywhere except these new-fangled containers...
Takes less space than a rename.
So I have zips littered over my filesystem with various datestamps.
And I still have to delete afterwards, so the terror is still part of the process...
My "why did Wilma have the rights to delete all that" alarm goes off. She should only have had write/delete rights to the one website she was supposed to update. A failure one step above, IMHO. And they should have implemented RBAC after that incident.
Though a few (20+) years back my mind was not so picky about who has which rights to prevent any disturbance of the infrastructure. Some things you learn along the reality. The university should have had that in place, but I know how it was "back then". Doesn't mean they are better now, but it should.
I had an SRCF site in the distant past and briefly checked before posting whether it was still around! It looks much more professional than it used to and is currently showing a message that they are looking to hire a system administrator so, while it is apparently still student-run, I'm not sure it's still solely student operated. It's what I meant by "in that form"; it used to be much more common for there to be some kind of student society maintaining a basic webserver on bare metal, SRCF might be the only one left. (Did a quick google for student run computing and other big UK universities and didn't find anything, one major university's CS society lists among their activities this year roasting each other's CVs and holding hackathons instead).
Aside: I was hoping there was a mortar board icon (even if Cambridge don't don't really wear them)
"It's a university. They don't have the time, nor the inclination, to do proper profile security."
Not only that, it's a Student Association. Probably exclusively "staffed" by volunteers, some for simple experience, some on power trips and some for course credits or CV/Resume filler.
Had a very hard time once to get rid of forest-level admin rights on a customer's system.
The project was finished, hand-over done, real end users on the system... and I still had access for months. No amount of begging made them at least deactivate my account.
IIRC we had to involve legal, who wrote a polite but firm letter that we couldn't be held responsible.
I also went to a somewhat convoluted process to ensure nobody knew the password any more (recruited three coworkers, initiated password change, each typed a word, a special character, and a number without the others knowing what it was; afterwards we all signed a statement what we did).
I use powershell for such things: Random generated 50 character string, made out of 8 bit "codepage 437" string, which includes nice characters like TAB, BACKSPACE, ESC, CR, LF, BELL, PAGE-FEED and all those nice "make a box graphic" characters beyond 128 - literally impossible to type... That is set a new password, and not stored anywhere. I regularly use that procedure for all deactivated accounts to ensure they don't get reactivated with an old PW we don't want.
1) Type the command but do not press <enter>
2) Take you hands away from the keyboard
3) Use your finger to point at each part of the command and check it's what you thought you had typed
4) Verify that you are where you are supposed to be
5) Commit the command
Also works with critical GUI-based activities, where you physically point at the various settings as you check them all.
People may think you look like a mad person, but that's much better than making a mistake.
To be fair to Wilma - We've all done it (or will do it). Showing the current directory in the command prompt is a standard out-of-the-box setting these days to reduce the risk, but back in the day it wasn't.
It's rite of passage that teaches situational awareness and paranoia. And the importance of backups!
That's where the elastic, bright orange bungy sprung taut between the chain or disc brake and the left handlebar comes into play.
All fine until somebody is pestering you and buts explosives underneath your train of through and destroys your processes. Had that just a couple of days ago. Topped up the air in the tyres on my car. Went through my ritural of not losing the tyre caps, blah blah, all done, final part is: open driver door, get tissue from door well, dampen it, clean hands, put used tissue into door well, get into car, close door. Whereup passenger stuffs damp wetwipe into my hand. What? What're you doing? Gerroff! Erughh. Now my hands are all wet, now I've got to open the door to get at my dry tissues to clean my hands again....... nearly losing door to passing van.
But it's Friday and half hour from knocking off time... you can stare at the screen as you go through the checklist and still miss the obvious.
Once had a configured router being shipped to a customer missing its vital ADSL card. 'What are we going to do to prevent this happening in the future?' Not much, I replied, as it would havet kicked up pages of errors when it tried to configure the non-existant card and great chunks would be missing from the config dump and final checklist capture.
(the guy responsible was extremely anal and unlikely to repeat whatever happened)
I think the better answer would be: "No late Friday afternoon production changes".
Instead, double check you've updated the documentation for stuff during the week.
If Friday arvo changes are necessary, then ensure that there is sufficient pause during the week, like some sort of team.building exercise.
I once learned the hard way that '<cmd> -r .*' in linux includes ../, and will apply <cmd> to the whole file system. Especially nasty when you use sudo on a multi user system.
After that, and a few other near-disasters with rm, I decided to precede every
rm <options> <my file pattern>
with executing
ls <options> <my file pattern>
And look at the output. Only then will I change 'ls' to 'rm' (or chmod).
And I also removed 'rm' from the history command, just to be sure, in ~/.bashrc:
export HISTIGNORE="rm*"
My bashrc has the following:
# It's supposed, probably correctly, to be bad practice to alias rm
# to anything less dangerous. I'm trying to learn to use del, but
# will not unalias rm for a long time yet!
# (I wrote the above more than 20 years ago. Is it a long time yet?
# jcb, July 2009)
alias del='rm -i'
alias rm='rm -i'
alias rm to rm -i
Works great until you have to delete a hundred files. If you know what your doing, calling /usr/bin/rm will override the alias. Don't just "unalias rm" - or someone else who expects the rm -i behaviour
may have a nasty shock.
Red Hat - this alias (and blue text on black background in a console) are just two of the thousand reasons I don't like Red Hat based distros.
I have to post this every time. If you do that, then, even if you never have a system without it, people will get very annoyed any time they do an rm -r, it gets turned into an rm -i -r, and they have to answer yes a thousand times. So to get around that, they'll do an rm -rf, and congratulations, not only have you taught them to disable your own command, they lose those warnings you get if you don't specify -f. Using -i all the time is harmful.
I do the same with any significant changes in the SQL data, and generally put the update or delete commands as comments so I have to manually select the code to execute it
SELECT xxx FROM yyy WHERE zzz
/*
UPDATE yyy SET aaa = bbb WHERE zzz
*/
Of course the very fact this forces me to think about what I'm typing generally prevents mistakes but just occasionally it saves me, usually 'cause there's something I've missed in the WHERE statement.
The problem is, you have to be personally responsible when using extremely powerful and dangerous commands.
If the problem wasn't typing "rm -rf" in the wrong place, it would move to hitting "y" far too quickly to any safety prompts.
Dangerous actions are dangerous, no matter the OS or the hand-holding.
Well, he is getting closer to accuracy - two months ago, he was claiming that Unix was written to use punched cards (of course, we all remember being frustrated by the limitations of Unix only *just* being able to handle 72 columns of text, after the sequence numbers and other special columns were taken into account, which is why, to this day, you always have to put a + in column 76 if you a C program to continue the line onto the next card).
No, Unix didn't have short commands and short error messages "to save paper and tape".
It had them because (1) Teletype™s are obnoxiously, painfully-loud, and (2) shorter commands mean less typing. At 10 characters per second (the standard Teletype™ speed), that's a significant real-time saving.
It had nothing to do with the noise of the teletype, nor saving paper/tape. It did, however, have everything to do with less typing. Brevity was the name of the game[0]. At one point, at Berkeley us lazy bastards even renamed AT&T's /usr to /u. Apparently it made things hard to understand for newbies, so we changed it back ... but I personally never met a newbie who made that claim. I suspect it was actually a professor outside the computer group who had "issues" ...
[0] Remember, absolutely nobody outside the typing pool knew how to type back then. Not even us always curious Engineering students.
I'm working on a Blazor app that clears a directory before users put files in it . The directory varies depending on circumstance and it picks up the location from its a database.
The other day ( in the development environment ) that filepath failed to come through so the app decided instead to just start deleting its OWN location!
It munched through the first layer of its source code in a sort of suicide attempt .
luckily the delete didn't include subfolders.
Being the sole authorI have only myself to blame.
Not quite the same, but I once took up an offer to do a wordpress inline update but wiped a custom template and made a terrible mess of the layout. I'm not into WP sites so only had a very light understanding how to make basic text changes, Fortunately the hosting company were able to restore a backup, but for a few hours I was working how much money I was going to have to pony up for a rebuild.
So can happen to anyone
Not a tale of deletion, thankfully. But once, my section found that all our files had disappeared and didn't know what to do - it was literal years of work that we were legally mandated to preserve.
We discovered with reasonable speed that one of our colleagues had tried moving a folder into a different subdirectory and accidentally moved the top level directory instead.
Into a subfolder within itself.
That required administrator permission to access files.
It even took IT a little while to sort that one out, due to the recursive loop.
I'd love technical details... Server/Client OS, filesystem, what the "top folder" only top on the share, or top on the filesystem and so on. My experience give following rule: Only admins have write access to the (NTFS) folder shared. All subfolders, at least on the first level, are configured to be only movable or deletable by admins, not by users. Only admin can change rights and onwer (and are the owner). For any subfolder below, if it has any explicit right, the same. Any folder with explicit rights is either 100% explicit or everything is inherited, no mixed rights (you can have reeeeaaaal fun with mixed, inherited from the folder above, above-above, above-above-above and so on. With the AAA variant you cannot even see where it got its inherited rights and what rights it got since you cannot see the folder above the shared level, you just know it does not work...)
My example is Windows here, but the same fun happens with nearly every other solutions too, unless you use FAT filesystem or something similar simple with no ACL.
"normally she'd just bring it back from Recycle Bin and that's all."
Depends. Very large files and/or vast numbers of files don't always go into the recycle bin. They may or may not be there. It's extremely risky to rely on that happening the way you expect. And then there's the vast number of "Are you sure?" prompts Windows gives users that they no longer bother to read them and press Y or click Yes/OK, so I'd say Windows was at least as dangerous for a noob or tired uses as any other OS.
Back in the mid-90s I worked at a place where we had 10 people to a Unix server for our workstations. (they were named after ores, I was on magnetite) I maintained a bunch of ksh scripts for our build and release management process.
One fine day i wrote a script for fun to see what would happen. Let's say it was called "go". It contained 1 line: "./go &". After making it executable, I ran it like this "./go &"
The process monitor I was running started to climb.... I quickly realized what was happening with my runaway script.
I heard my colleagues on magnetite start to grumble, and my shell was ve r r r y s l o w. I thought the server would crash, but it plateaued. I had to own up to it, and got our Unix admin to my desk.
The PIDs were changing too fast to find and kill them.
With 9 others standing around my desk, the Unix admin said he was going to go reboot the server, which meant people would lose what they were working on. As he walked away, I still had my prompt and said "wait!"
I typed one command, and it stopped. What was it?
.
(I've used this on interview questions before, and nobody has gotten it)
.
.
That command was "rm go".
One of my personal favourites, from an ex-colleague.
At a place he used to work they were old skool, having just the one server for both test and live, with separate environments.
One day, they were running very short of space and since a release had recently gone live so the test environment was no longer in use, it was decided to remove the test environment's data until more disk was aquired. This resulted in the following exchange:
"Er, ${regomised_name}? Is there anything you'd like to tell me about the test Customer table?".
"Oh, right. We were short of space and we were only reading from it, so I hardlinked it to the live one. Why, is there a problem with it...?"
…. For the actual sysadmin to do some knowledge transfer. Deleted that? Ok here’s how we get it back and you can sit here and type these commands.
Now what were we trying to do? Let’s walk through that so you know next time.
One day Wilma will be in the wild with a Who Me? experience all of her own (or someone else’s) making and she’ll know what to do.
It was a time before Linux, even before Windows... Year 199x, me 20-something, just got my first IT job...
I have just read (and re-read, and re-re-read) and tried all (well, most) of the examples from ~12 brand new red books titled "Novell Netware 2.2", especially the volume 1: Installation. Before that, I had experience with MS/DR DOS, fdisk and such, but creating a non-dedicated Netware server was a new thing. Somehow, a customer's server crashed at just the appropriate time (with no backups available, of course), so I was given the task to bring it back from dead, or at least the data. Server was hosting CA Clipper (Summer '87 if you must know) based "database" files.
So, I did the most normal thing (under DOS): installed a new disk in a new computer (back then, we used to assemble them from parts), also installed the disk from crashed server (because it tested physically OK, just wouldn't boot), hoping to just copy the data from second drive once the install was over, and started the Netware installation. Now, Netware 2.2 installation had a very unexpected "feature": it removed all of the partition info from all attached hard disks without asking anything, Only later it allowed installation of DOS if you told it that you want a non-dedicated server (sorry if this sound confusing, this is how i remember it, books are long gone, and I didn't consult any AI on this).
Next thing I did is phone my two (much older) colleagues, explain what I did, also told them about this excellent new program that we got a few weeks before called "Disk Editor", so we spent next hours reading disk sectors, copying database sectors from disk into files, and turning resulting files into appropriate .DBFs. We finished just after midnight, disks were much smaller in those days, and we were lucky that the server contained only this database, so everything that looked like a DBF was a DBF.
Take a snapshot backup and then make changes. You can always rollback if it breaks things.
Also my webhost being 11hrs behind in the UK means I have often solved this issue myself by the time they get back to me!
My own website may not look too flash but after 22 years I am used to it and it is a backup as I have seen platforms rise and fall.
I also deleted a web server once. It was a learning experience like no other. It all kept running, because files that are in use will not be deleted immediately. I recreated the whole system in a new virtual machine, then switched over. That was a verrry looong wasted weekend…
Nothing like the "white terror" as the blood just drains from your body when you realise that the command you just typed is running and it's too late! After 40 years in this game I have a rule, anything run as root that remove anything, hands off keyboard for a few seconds, re-read command before hitting enter!
I had a colleague who deleted our web server two or maybe three days in a row. At first we had no idea what was going on, then I found the cron job he had set up to do something very different. He had forgotten that when his job was run by cron it would be running in '/' and not the directory he was intending to clean up. Whilst embarrassed by his mistake he was more than a little relieved to have found out why his cron script had been failing to clean up the intended directory.
>Linux command line
The only command line Linux has is to set bootup options (for those with good OS's, see GRUB_CMDLINE_LINUX in /etc/default/grub).
It seems she was using GNU bash and GNU rm, but we cannot allow the users to even learn of the GNU/Freedom, so they forevermore remain; https://www.gnu.org/gnu/gnu-users-never-heard-of-gnu.html
I recently left a position that involved a significant amount of time bouncing around in many pieces of network equipment spread across a large geographic area where virtually all sites were unmanned and most were uninhabited and made the planes, trains and automobiles movie look like a trip to the local 7 Eleven. The person that had the luck to take it over sent me a picture of a large Post-It on her monitor that said "WHERE ARE YOU?" that was installed after a rather unfortunately urgent trip to an unfortunate place using unfortunately expensive modes of travel. Luckily, the network devices all have some sort of try before commit modes or features and that lesson was learned early. Many other devices at the sites were not so lucky. Also, I don't have tons of files and folders with names that look something like name-, name-- or name.oem I would never do that.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
