The sad tale of the Alpha massacre Good morning and welcome, once again, to Who, Me? in which Register readers share tales of tech support moments they might prefer to forget. But forgetting is not a way to learn from mistakes, is it? This week's hero is a veteran we'll Regomize as "Gandalf" for such is the grayness of his beard nowadays. Back in less gray …
These days, of course, rm -rf / comes with a "are you really really sure?" notice or even "I can't do that right now, Dave" messages.
However sudo dd -if /dev/urandom -of /dev/sda does the job, some would argue, more efficiently/thoroughly and with no safety net.
I suppose you are less likely to type sudo dd... but now you know, if you didn't before, its going to be hard to resist at least once!
Icon cos I love seeing things burn and it was the season for that just recently!
I once did the ancient equivalent of this, on an IBM System 370. What I was SUPPOSED to do was copy the disk containing all the operator "procedures" to a backup disk. In those dark days, a "procedure" is what today you would call a command script, and the operations staff had a big book of scheduled things they had to do, like run the company payroll, or inventory updates, or something, and the book would tell them to "Mount tape number 2375 and run procedure XYZ". And a disk pack, weighing ten pounds, could hold about 250 Megabytes. With an "M".
But I got the device definitions backwards and copied the empty disk on top of the the live one, erasing it. Under the glowering gaze of my boss, I went to the tape library, fetched the most recent backup tape, and copied it to the proper disk as the operator staff stood there waiting. No timesharing here - this computer system that filled a room could do ONE thing at a time.
When I was in Support, we had a large well known shipping company dd the root disk to a backup disk the wrong way one weekend. Tape recovery obviously needed. The only reason for this comment is that they failed to change the script and did the same thing again the following weekend! muppets...
As a kid, my friend loaned me his copy of 'White Lightning', a piece of software for making 'backups' of Amiga discs.
I managed to backup my game alright, right onto the disc holding the copy program. I tried to blame it on my friend for leaving the disc with the write-allow-tab enabled, but I knew it was really my fuckup.
RHEL aliases rm with a warning as rm -v. It is best to pay careful attention to those prompts.
In this particular case, a cleaning script should have been used which included:
[ -n "$QATOOLS" ] && rm -rf $QATOOLS/bin $QATOOLS/etc $QATOOLS/incl $QATOOLS/var
On many Linux distributions, the "ReaR" application can be used to make a bootable restore USB that will format and rebuild a lost system. Very handy (and surprisingly written entirely in the bash shell).
On the other hand, I can see how, if there were such a thing, there would be times that the -n was intended, but somehow didn't get into the actual command line. Which would have consequences, of course. The cure just might be worse than the disease, in that case.
That last part reminds me of a long, long time ago.
I read that on Linux, rather than reserve a whole, fixed-size partition for swap space (and those were *hard* to resize, given you had to resize and move the filesystems around them..), you could store a swap device on your filesystem!
.... Hmm. I thought, lets give that a try.:
# mkswap /dev/sda
# swapon /dev/sda
-- and I was quickly greeted by a kernel panic. Oh, for the things we learn. (`fallocate --length=2G /swap.file; mkswap /swap.file; swapon /swap.file`)
One of my carry-overs from my first DOS 6.22 & Win 3.10 days, tweaking my exec.bat files, is having a few aliases using what was then the dir command, with options and increasing levels of detail for more 'd's in the alias name. So, d for ls -F, dd for ls -FlhA, d for ls -Flhani --full-time. (I don't remember just what the options are in DOS-land, and these days, it's actually using ls-F, because tcsh.) This would seem to be absolutely terrifying given how often I'm using a system not my own and that might not have my aliases set up on login, and I could just dd a directory at any time. But, given the unusual syntax of /usr/bin/dd, with everything aside from what, --help and --version requiring being declared with an equal sign in it, anything I've ever done a directory listing of would result in a syntax error when run with /usr/bin/dd anyway, so not long after I switched over to Linux and noticed the collision, I realised it wasn't really the problem it seemed, and got over it. (When I need the actual /usr/bin/dd, then I just spell it out with the full path, and the alias is ignored.)
ETA: As with the commenter before me, I'm not familiar with OP's -if & -of syntax. I know the one where it's all equal signs, as implied previously.
had that issue on AIX. xargs helps. Other way was shell expansion rather than find or ls. Both stopped terminal dead for indefinite periods. Shell expansion worked as empty junk dirs were created with a predictable pattern so a few loops cleaned them up efficiently.
I did that as root, and setting all ownership to root:system on our test UNIX domain (LPAR) that ran on an Amdahl mainframe running MDF many, many years ago.
But I was fortunate that the test environment was sufficiently close to the main one that I could grab the correct ownership of all the files that were common to both systems (which was most of them as the test environment was not a complete copy), and restore them using a quick bit of scripting, and also fortunate that I had the test environment booked for exclusive use, so nobody noticed, and I managed to get it all put back before I had to let anybody else back on.
As far as I am aware, I got away with it.
yes, IIRC correctly, making rm recurse without the -r. Had to debug a "cleanup" script that took out a prod server once. It had run for a while without errors but left hidden files alone. These also needed to go from a scratch dir which was filling up with them. Junior just added .* to his rm script run from cron as root. He was lucky to not be fired.
> How many of us have typed "rm -rf *"
Not quite. But one time (and one time only) I accidentally inserted a space in the worst possible place when clearing out a directory.
Here's what I meant: rm -rf /tmp
And this is how it turned out rm -rf / tmp and it goes without saying this was all done as root.
Whose bright idea was it to make the space bar so big?
Here's what I meant: rm -rf /tmp
And this is how it turned out rm -rf / tmp and it goes without saying this was all done as root.
I too did something like that with my first Linux install (Slackware 0.99pl15 in the very early 1990s on a mighty 386sx25, 4MB RAM and a 330MB ESDI drive that had to sit outside the case because it was physically quite big.. Acted as a good room heater though).
Just finished setting everything up (FTP'd a bunch of scripts from Demon to do dial-on-demand, configured qmail and NNTP). At about 2am I realised that /tmp was about 100MB in size so did the dreaded rm -rf / tmp
Then hit ctrl-c repeatedly to stop it before everything got nuked. Fortunately, the /etc directory was still intact so, after copying it elsewhere, rebooted from the install floppies and rebuilt it. Didn't get much sleep that night.
Learned a valuable lesson - when doing destructive stuff, check *before* hitting enter!
sounds like my first Linux install. 386sx/16 4 GB ram. made by Dec. only had 4 good floppies so would download the first Slackware disk then image the floppy just in time to insert it as prompted by the previous floppy the IDE disk was 40 MB on the machine i was using but much larger on the dec. set it up as a web server just as a test. eventually became the real thing. used to compile Linux but so slow that i did it overnight.
I tend to put a trailing / on all my directories which can mitigate this sort of thing (unless already in /), thus (in bash) there would be the possibly missing
QATOOLS="/opt/qatools/"
and then
rm -rf ${QATOOLS}bin
Also, it just occurred to me that (again in bash) you could even try to remember to type (or script)
rm -rf ${QATOOLS:-QQQQ}bin
so that the rm shouldn't get anything ever, unless you really like directory names with Q in them:-)
Yup, knew immediately when I read that line what was about to happen...
Always, ALWAYS, wrap up procedural things like this into a utility script, no matter how simple. Never try to wing it each time because you will fuck it up eventually no matter how good you are (or think you are).
When the script messes up due to unforeseen circumstances you can strengthen it ($?{VARNAME} syntax etc.) so the next guy is better protected.
("rm -rf $UNSET_SHELL_VARIABLE/*").... and deleted many punters' saved games, stats, in-game awards, and such.
Anything like this which is going to be pushed out to world+dog needs a code review, ya cheap bastards!
(I had a Steam installation on a hard drive of my gaming PC. Fortunately for me, I had it switched off all that week, and Valve did pull the update after the complaints avalanched through their offices.)
I had to delete a directory with a name like &NAME&, no problem use backslashes, but muscle memory took over..
rm -rf /&NAME/&
Luckily I was not root, and the machine was slow, so I could kill it (warning messages gave me a clue). Dev machine, not too bad to get back together. I now use double quotes
Tip: tab completion will make the shell fill in the escaped characters correctly, and works with just a \ to distinguish "hello world" from "hello" by adding a \ after the hello and pressing tab.
It will also flag a lot of other errors by virtue of not working unless it matches an actual file, so is a useful habit to get in to.
I'm not saying it was good form, but sometimes I was just too lazy to make a /temp directory to unzip a program installer into, so I ran it from C:/
Mind you, I wouldn't let pkunzip fill my root with scores of files; this would just be to uncompress a monolithic executable installer.
Until I met the one which installed it's payload and then cleaned up by deleting everything in the install-from directory. You know. Everything.
Back in either 1989 or 1990 I was visiting a friend, and he wanted to put a couple of files on a floppy (yes, a 5 1/4", following on from the discussion about what disks could legitimately be called "floppy" from a couple of weeks ago) for me to take home. He popped it in, did a dir a: to see that there was nothing important on there, and then a quick del *.*. As it took a few seconds to complete I could see the colour draining from his cheeks. He was on his dad's PC, and had just deleted all the files in his C:\ directory...
Luckily, his dad was an organized person so the only things in that folder were the things necessary to boot up, all work files were properly tucked away in subdirectories, so the biggest pain was recreating AUTOEXEC.BAT and CONFIG.SYS, including finding the right control characters and codepages to get his printer to print Scandinavian characters again (CP 850, echo Alt-135 (ç) > LPT1: in case any archaeologists trying to do just that find this post.). Luckily this was before either of us had a Soundblaster card (PCSPKR.DRV FTW) so we at least didn't have to go interrupt chasing.
Certainly back in those days, most people learned quite early on not to put loads of stuff in the root director of MS-DOS. The root directory entry had a relatively small limit on the number of files it could cope with. Sub-directories could handle far, far more files. But yeah, most people still put important stuff in root :-)
Luckily this was before either of us had a Soundblaster card
Got my first Soundblaster card as a freebie with OS/2 Warp 3 in 1994 - the bundle was OS/2 Warp 3, a Soundblaster 16 card and a CDROM drive. Also managed to blag a free OS/2 t-shirt (which, ironically, I chose to wear today. It's looking a little ragged around the edges, much like the wearer).
Plugged it all into my shiny new 486 desktop and, half an hour later, was running the glory that was OS/2 Warp. With sound!
Unless you're using a cheapo no-brand SoundBlaster clone, usually the Address is 220H, IRQ5 (unless it's an earlier issue SoundBlaster card which uses IRQ 7 which does conflict with the sole Centronics port in the system), DMA 1, and full-duplex cards like the SB16 and SBPro 2.0, HDMA 5.
if it were on a tru64 system QATOOLS=/usr/opt/qatools as I recall /opt was reserved for third party kernel packages.
Alway a startlingly good idea to script these things... even more startling if you don't as this then young wizard discovered .
Presumably his employer's database product was packaged using the native tools for each supported platform - setld kits in this case. I would have thought packaging the qatools software etc would have made sense as even setld -f -d ... couldn't come close to rm -rf /. I recall initially creating kits was tedious but definitely not rocket surgery.
Oddly enough I seem to recall Informix being one of the applications on the Associated Products CD.
which is why my scripts did a check of all required variables and settings before it did anything. If critical vars were not appropriate size script stopped with error message. I usually assumed scripts would be be run by successor admins of dubious competency. Worst case I had as middling experience admin was was an academic had a backup of all their work on a tape using tar, which they wanted to start from in the new $CAREER. Every file a single tar archive. By the time I worked out there was a couple of Gb files on tape which required use of mt initially then use of the norewind device to loop thru the tape I forgot which directory I was in, naturally it was tadaah /. The amount of extra files that needed to be moved taught me to do pwd before any tape read or write and scripts had an ENVIRONMENT section where variables were set with correct or safe defaults. At stable sites there was an agreed variables directory that all code that needed them could source. This meant far fewer running code changes with easy rollback.
Once, a long time ago when I was a relatively inexperienced software engineer, I needed to clear the contents of the /tmp partition on a Sun-3 workstation running SunOS (yes, it was *that* long ago). So I logged in as "root" and typed the immortal command "rm -rf / tmp".
Note additional and very much unwanted space in the command.
An "oh s**t" moment occurred immediately after hitting return followed by a lighting fast "Control-C". Lightening fast, but not fast enough - I spent the next couple of hours rebuilding the system from the installation tapes.
I tend to put a file named -i in directories a script might find interesting. Forces rm -rf back to interactive mode. Not foolproof of course but slightly better than nothing.
Did rm -rf / opt once, nearly got fired, and not have a tingly feeling down my spine whenever I put any arguments after rm...
Ah, so we should use the double hyphen, i.e. ....
rm -- *
... but then why create a "-rf" file in the first place?
[edit: I suppose you might do it by accident, with one of those misplaced/mangled pastes that create a load of files named after the commands you wanted to run. Or is that just me?]
Not quite as bad as that - just the directory I was in and all subdirectories. Bad enough that it was all of our software on a remote production system in Brentwood (over a 56k modem connection) - the customer in question did notice. The rather more experienced boss got me out of the shit that day.
Though I am less of a Gandalf than a non-scottish Rab C Nesbitt.
The first one was on a Sun box. As a greenhorn I had followed some instructions to put /usr and /opt as well as /var in different disk slices. The time came to construct the data partitions on the attached array and I entered newfs /dev/rdsk/c0t0d0s3 rather than /dev/rdsk/c1t0d0s3 and I watche /usr disappear. Thankfully it wasnt in production so the three days of rebuilding was just a good learning experience for me.
The second was a good few years later and the Local Authority I worked for had adopted this new fangled Linux for some 'non-critical' stuff. This particular toy (even on vmware by then) did all the interface file transfers between systems so was clearly non-critical </sarc>. Anyway, for some reason I needed to put some files into a directory in /etc. It arrived as a tarball and without thinking I let tar xvf destroy the whole of /etc. It was remarkable how long it took to fall in a heap but I still had days of desperately trying to get it to a state where I could get the restore software to talk to the backup server before finding out the backups had not been working (it really was a self inflicted perfect storm). As I remember there was little come back as manglement didn't really know what I did and nothing too critical like the benefits BACS file still went properly.
In the dawm of time I downloaded something called "linux" onto recyled NT3.1 floppies donated to our school by Microsoft. Rushed home installed for several hours. Acheived root.... typed: "mv / /tmp" ... and things got weird fast. Think that bug was fixed somewhere in the 0.98 kernel release.
I looked at their process and figured a script was needed.
Absolutely but that won't stop those who think they know what they are doing from proving they don't.
If the shell had errored on finding $THING empty, or had errored on encountering the subsequent "//", no harm would have been done. But input validation is for sissies, right?
from yet another old destroyer of /
How about, as a safer alternative, starting out with:
cd $QATOOLS
rm -rf bin etc incl var
(and, if you haven't invoked pwd in your prompt, then you can add one just after the cd)
A lot less typing, no daft putting a / in front of bin etc (etc) and risking killing off the key OS files. If you didn't spot that the cd "failed" then, depending upon your shell, you either just went to your logged in user's home directory or simply stayed put (which isn't, of course, in the root directory - please!). Both options hopefully far less damaging to the system.
Of course, all the other tricks can also be employed (see above, doing ls, using a script...)
I've done that. We had personal Interactive Unix workstations. The owner of another WS asked me to remove the account I had there to free up space. I deleted my /home/me but left my /etc/passwd entry, changing the home field in there to / so that I could still log in. Unfortunately the owner then proceeded to invoke the remove user admin script blithely accepting all the "are you sure?" prompts. Well it certainly freed up space. 15 5¼" disks were needed to reinstall.
If you run a script (or use an alias) 'clean_qa' then these types of screwups are impossible. IMHO scripts should be used for anything potentially destructive where getting things wrong can either result in lost data / restoration from backup or system downtime / need for a rebuild/reinstall.
You can also have the script check stuff not mentioned like if the database is running, if you have the necessary permissions and so forth for further safeguards. If someone manages to screw up using the script you can add another safeguard to catch that case and at least insure that particular screwup can never happen again.
It is hard enough to make sure YOU don't forget one step just one step when you're rushed or hungover or whatever, but when you have multiple people who may need to do such tasks and maybe some of them are a bit more careless or green than you, it is dumb to just rely on manually typing things exactly correctly without a typo like "set QATOOL=/opt/qatools" that would be the same thing as not setting QATOOLS - or maybe you have an 's' key that works only 99.99% of the time.
Just to share... I've done way worse: back in the mid-2000s I was building a series of web server 1U systems over the weekend at my company's offices to be installed in a DC the following week.
At some point, after many hours of fiddling to get things working, I wanted to delete my work data and executed the command "rm -rf *" as root at which point, things started to go awry.
Turns out, in my sleep-deprived bleary-eyed vision, I had auto-piloted this to "rm -rf .*" which wiped the entire filesystem.
I was quite impressed though: quite a lot of stuff continued to run.
I then had to waste another whole man-day on rebuilding the thing. Oops.
Login as room
Sudo to user
Hit Cntl C as some point
cd /
rm -rf *
Done on a production system (A temp system was on the same box I was trying to delete)
. Thankfully after hours. These were the days when you had to construct your
own file systems. High access DB files on the outside of the disk.
A very long night was had rebuilding the system before restoring the database
We had a similar incident back in the day ... I worked for a property management company that managed shopping centers and we ran on a weird UNIX variant on weird UNIX micros from Fortune Systems. Each department had one ore more of these multiuser machines. In MIS (oh, those were the days) our Director logged in on our machine with superuser rights and rm -f his home directory (the command was slightly different but had the same effect). Wondering what had gone wrong he went and nuked a number of other home directories. Then he let the rest of us know that "something was wrong". All hell broke lose and much work was performed to recover. Certain passwords were changed and the Director was locked out of all systems (the Unix boxes as well as the Prime mini's); he was left with his primitive golf game on his desktop PC.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
