Backup script ingested an accidental asterisk and deleted everything WHO, ME? Welcome to Monday morning, the time of week when The Register always asks “Who, Me?” because that’s the title of our reader-contributed column in which you confess to having made a mess, and found a way to egress without career distress. This week, meet a reader we’ll Regomize as “Miller” who told us that as a …
The fact that that data could be restored shows that they were not totally clueless. A mistake was made, but a fix was available. Today, with all of the modern practices, devops and such, it seems that backups and restores are not always properly implemented and working. Or even just considered.
In 1981 we understood the need for such things, computers were less reliable and we didn't assume that somebody else would take care of any problems. I well remember Monday mornings as a part-time admin, booting the VAX under standalone backup and copying the "huge" 256MB system disk to the spare, with the source disk going into the fireproof safe afterwards. Could be nerve-wracking, it was always a comfort to press the write-protect on the source drive. I know one admin in the company who didn't do that, and then reversed the source and blank drive names in the backup command...
"And I am super extra careful... so I did it only once."
Polite suggested correction : should really be "And I am super extra careful... so I did it only once, so far".
Until one is pushing up the daisies, one can never be sure of there being no repeat. The oft quoted here Terry Pratchett meme of once in a million....
I once made that mistake on a disk drive with an integrated cartridge drive which had a great feature that allowed you to clone the disk to the tape with a single button push, or vice versa. Which is great right up until the point you accidentally do "vice versa" with a blank tape...
@Kurgan
Yes to write protect.
About forty years ago, my employer promoted a guy from the loading dock to the job of running update tapes for customers. The fellow who was his mentor insisted that he bring him the write wring from each tape he cut, which the mentor would toss into a drawer. One of us once asked, Gee, Ham, what do you do with all those write rings? Ham said, I give them to the orphans, pal.
Admittedly not thanks to code he had written, but I recall one of our interns managing to move several TB worth of data off a SharePoint server he was responsible for into the recycle bin.
He spent the next ten or so working days digging through the recycle bin restoring everything back to its original place. I think having to use the horrific Sharepoint web UI for that length of time was a fitting punishment in itself.
I've seen this mistake multiple times. you make a string out of root path + discovered target path + "*" and then feed it to rm -rf. Now if discovery fails and the discovered target path is an empty variable, you get a nice rm -rf /root/path/* Or maybe just rm -rf /*
Easily fixed by MULTIPLE REDUNDANT error checking, and a sanity check in the end like "if the string's total length is less than X, abort".
I've seen a windows program install / uninstall script delete the whole drive (in win xp times) because of this.
Not entirely sure what you are protecting against.
If the purpose of the IF test is to ensure that you successfully changed directories then testing for success after doing a PUSHD $SOME_OTHER_DIRECTORY would be simpler.
One advantage of the IF test is that if SOME_OTHER_DIRECTORY is soft linked to somewhere else then it will fail whereas PUSHD won't.
Why would it default to allowing a SCRIPT to execute a wildcard delete of every drive in the system without any sort of confirmation? Would you expect an OS to allow formatting the system drive the OS is currently running on without confirmation? Neither of those are likely to be things that you intended to have happen, and if you do you'd do it so rarely that having to provide manual confirmation is not a problem.
Now yes he could have sanity checked and made sure the drive letter he was deleting was in fact a "letter" or running some sort of command against that drive letter that might have uncovered the issue. But while I might have been curious "what happens if you run out of drive letters", unless there was some sort of OS documentation you could read that would have told you my assumption would probably have been "you get an error message if you try to use a 27th drive letter" not that it assigns a "*". Did every drive past the 26th get the same "*"? If so, how would it have been possible to uniquely identify drive 27 vs drive 28? Was there some sort of index number or UUID he could have used instead?
At the college where I was teaching someone was deleting all the files in the homework submission directory. Nothing was working. I tried submitting something and left the program running. I then went down to the support person and asked him to print out all users logged into the course submission account. My name came up and I seemed to be running rm -rf * in the top-level submission directory.
I stopped my program from running and after some trouble shooting discovered the problem:
The submission directory was over quota and the submission program was unable to create any of the sub-directories needed for the submission. The submission cleanup script then did a rm -rf * on the top-level directory because the system had failed to create the temporary directories so the cd commands were noops.
Once we found the root cause we just doubled the quotas for the submission directory making the problem go away for awhile.
Next day I told the class what had happened. I mentioned that backups would be useful in this case and the class chuckled. I then explained that we had a fairly extensive backup system and could (and did) restore the files from a few hours before the problem. Students did have to resubmit any homework from the previous day but not a big deal.
I wonder which alphabet they were using
Dunno about this "mainframe", but DOS typically allocated A: and B: to (non-existant) floppy discs and C: to the main hard drive. Z: was often allocated to a network share. Additional drives therefore often started at D: and finished (at best) at Y: so, Miller plus 20 others makes 21, plus A, B, C and Z makes 25, leaving just one letter of the 26 "free" until a new user is added?
M.
“The routine attached to all users’ drives and backed them up to a temporary drive,” Miller explained. “But you never knew in advance what drive letter the system would assign to the temporary drive."
Which I read as mounting each user's drive *plus* creating and mounting another one to back them up to.
The "20 others" thumb twiddlers might not meant literally - 20 odd, perhaps. Maybe a few more intelligent of the crew decided to bugger off for the rest of the day to do something more productive. In 1981, in some places it wasn't completely unknown for employees to be found propping up bars during work hours.
The basic programming lesson everyone needs to learn, sooner rather than later, is to always check for error returns - if the return is not OK and you don't specifically handle that error… bail. Only a fraction of cases it might be better not to abort but deciding those comes under the heading of specifically handling if a little anaemically.
DOS typically allocated A: and B: to (non-existant) floppy discs and C: to the main hard drive
Having used a DOS machine with 2 floppy drives (fancy, wasn't mine) of the 5.25" variety in the early 1990s (roughly a decade after the story in the article) I can assure you that A: and B: weren't always non-existent. Neither was the existence of a hard-drive (expensive!) a given in this period.
It vaguely sounds like IBM's mainframe OS CP/CMS aka VM/CMS, which was current in that era, where all users ran in a virtual machine with virtual disks, card readers, and card punches (yes, card readers and punches!), though the disk naming and handling conventions were considerably different.
Not many years ago, I changed the motherboard in my "Legacy" m/c.
Thort I'd load it with old disks: 2*3.5" plus a Zip ( remember them?)
Turns out all my floppy cables (actually the stiff sort) I had cropped to single point to point! Arrrgh!
Found a computer shop (was a few years ago) with an old fashioned ribbon jobbie.
Plugged it all together and found the "new" motherboard only drives ONE floppy! B***er.
P.s. Philo -- Any relation to the old boy I used to ring with name of Farnsworth?
-- or the chap who invented TV?
When HDDs were too expensive, having 2 FDDs was very handy.
Not sure reserving drive letters is 'wasting' them so much as sensible design. Ditto MS's endless 'Do you really want to hurt me?' text queries/dialogue boxes when you are about to do something. Even Jedi make mistakes.
That's 2 nice things about MS, so I've maxed out my quota for the rest of the decade.
Yes. I remember an old PC with 2 floppies, one of" each" size (5.25 & 3.5) as it was the time when 3.5 was becoming dominant*, but 5.25 had not been totally replaced & a period of transition.
.. making me feel old!
* as more storage, smaller size (though thicker) & protective shell were big advantages of the 3.5
DOS typically allocated A: and B: to (non-existent) floppy discs and C: to the main hard drive
Not on all DOS machines - I think that was more a function of IBM compatibility.
ACT allocated the disk drives in order of HDs then FDs so an HD machine with a single floppy would have A: as the HD then B: as the FD but a twin floppy machine would have A: and B: as FDs.
Which was fine if that's all you were working with, but if you'd just come from a PC compatible and went to format a floppy when it was all too easy to type FORMAT A: <Return>, quickly followed by "Oh shit" and Ctrl-C! Thank goodness for Norton UNDELETE!
Had that at a company many years ago, A contractor was in and wrote a quick CMD routine to do some work on some cluster nodes. I had no idea and even less idea why the guy who looked after the clusters had spent the last few weekends in fixing them until I get asked to have a look at a script that is a bit suspect. On initial checking I see nothing obviously wrong. It is just tidying up the folders it was meant to do.
I them take the script and run via in step mode and yup, all good until the last line. It asks me an unexpected question so I stop. I had another look and there was an extra space at the end of the line that made the command want to delete the whole drive - which is what it had done for the last few weekends.
The command runs nicely without quotes around the folders, but did not make this extra space visible and the ramifications it held
had another look and there was an extra space at the end of the line that made the command want to delete the whole drive - which is what it had done for the last few weekend
Many years ago, (early-mid 90's?) when I was playing with linux for the first time (all those floppies - downloaded by a friend at work because they had Internet and I didn't.. Slackware 0.99pl15) I had, finally, set everything up including dial-on-demand to Demon (remember them? At the time they were the techiest of ISPs and had FAQs on how to do stuff like setting up a mail server/dial on demand etc for linux).
So there I was, with a fully-working linux box, patched, one ping away from internet access (wwoo-wiing-pang-pang-swoosh[1]), at 3am. With a nice glass of single malt to celebrate.
Thought "I'll just clean up the tmp directory"..
Typed "rm -rf / tmp" and hit enter.
Then frantically hit ctrl-c until I got the command prompt again.
Fortunately, all the relevant bits (/etc mostly) were still intact and the slackware installer let me do an upgrade install without nuking stuff.
Lesson still very much stuck in my head. Don't do potentially-destructive stuff at 3am, after a (ahem) decent-sized glass of whisky.
The call for temp disk failed and my routine passed back an asterisk instead of an error code,” Miller confessed.,
WDamn unlucky to be fair , what an unfortunate sequence of events !
Theres been an error - here have a wildcard and play with that - it might do the job!
Its kinda similar to how a "Buffer overrun" will execute an inturders code for them , rather than brick the machine abd crash it - which to me seems far more likely
Sounds like a young graduate who I had the misfortune of working with in 1988/9 on a very basic IBM PC (I think it had a whole 20mb HDD).
Junior had a bad case of Dunning Kruger where using a computer was concerned and on several occasions wiped key files off the HDD due to a combination of sheer incompetence and not doing the basic RTFM for DOS commands before hitting return.
I think I bailed him (and the project) out on no less than 3 occasions because I'm the sort of obsessive who backs up everything (in this case on a box of floppies securely locked in my desk drawer). On the third time of him wiping a weeks work (no waste bin in those days to retrieve the files) I threatened him with physical violence having had to spend good drinking time on a Friday evening sorting out his mess. Junior didn't drink.
The threat of violence got me a telling off albeit a gentle one after I explained what he'd done.
There was the special treatment for traitors of being hung, drawn and quartered.
icon: Guy Fawkes
This post has been deleted by its author
You know what would really impress me as an example of AI? Capability built into the OS that could spot something like this and go "I suppose it's possible you intend to erase everything, but it seems unusual. If that is really what you meant, [do some elaborate procedure much more complicated than simply clicking "Yes" in a dialog box".
Of course, this would probably require a _real_ AI that understood something about systems and ops, not a stochastic parrot that had reduced the entirety of the internet to pink slime.
Where as current AI would go "What a great script. You're so clever, I'll delete everything for you right away. In fact I know you are so intelligent, you probably want me to wipe the BIOS as well, so I'll make sure that is done as well. No need to press OK as I've done that for you, because a genius like you can't make a mistake."
