Techie pulled an all-nighter that one mistake turned into an all-weekender Well, would you look at the calendar? It’s Monday already, and by lunchtime any fond memories of the weekend will have been erased by work worries of the sort The Register celebrates each week in “Who, Me?” – the reader-contributed column that tells your stories of making messes and somehow escaping. This week, meet a reader …
Sleep deprivation does funny things to your brain. The other day i had a narrow escape, i ran a command i was going to pipe to a force delete, but because i wasn't 100% familiar with it i wanted to check the output first.
...30 seconds later...
hmm, this is taking longer than expected, what's up?
Turns out i typed /* rather than ./* i guess that's quite a few files to go through...
I once cut instead of copied a folder in a directory and posted it into my new folder in another drive. After I’d done this I was about to go back into the original folder and leave a read me text file. However the folder wasn’t there, and a realisation that I might have clicked cut rather than copy dawned. I replaced the folder back where it should have been just as my desk phone started to ring. That was an confused rather than angry user who wanted to know why their software wasn’t working properly. I suggested that they just restarted the thing and maybe the issue would go away.
Same.
The lesson learned is: ALWAYS put a `/` at the end of the _directory_ that you're moving things to. Develop the habbit that not having the / at the end means that you're renaming the file/directory, not changing its location.
and people say, "You know you don't have to include the trailing slash, right?" Knowledge is looking for oncoming traffic when you cross a one-way street; wisdom is looking both ways, anyway.
I once had to confirm with a coworker in my home country that the plan of action I had just come up with for a customer support team after arriving in Taiwan in an exhausting 30 hours of travel with little sleep on the plane actually made sense. My brain was working sufficiently to form the words and I was pretty sure what I was saying was correct but somehow my brain was also so frazzled I just couldn't be sure that what I was actually the right thing (it was, but still).
The rule to cap them all is that when the big main task if completed, and you're knackered, you take a break, refocus and only then go back to do the tidying up. This applies in every field of endeavour. Three reasons (there may be others)
1)Your mind will still be focussed on the original task - so details in the subsequent activity will be lost
2)You will be in a hurry to get the Hell out, so pausing to think/let alone double check, will be passed over, if only unconsciously.
3)If you’re tired your concentration will be shot. You are going to make mistakes, forget things or just miss signs.
As noted, not just in computer things. Anything where a tiring substantive task requires some tidying up after completion.
Exactly. Years ago, we had a major production outage that I think might have made El Reg. The root cause turned out to a ImageMagick bug in a previously installed application that suddenly started getting heavy use, completely melting the CPUs. Because the app had been installed months ago and nothing recent had changed, it was difficult to find.
Once we did find it, I was completely knackered. I asked a co-worker to install Apache on some new servers so we could move the new app to them, freeing up the CPUs on the existing ones for the main app. Normally installing Apache on a new server is something I can do in my sleep, but since I hadn't had any, I needed my co-worker to do it. While all of this was going on, our system admin wanted evaluate SAN performance. I said "No, we identified the issue and that isn't needed. *My* CPU is fried, and I am going home."
I don't trust any system to have that safety net in place to safe me. My approach is never to do a "rm <pattern>" directly but first use "find <pattern>" and if the result lists those (and only those) files and directories i expect i append "-print0 | xargs -0 rm -f" to the find command.
dd needs triple checking - way to easy to zero out your hard disk instead of the flash drive.
I and my hard drive have met that friend! :-(
Since then I've had a dedicated box with 2 hard drive bays/hot-swap removable drive-carriers, a CD/DVD reader, a CD/DVD burner, and no internal hard drive.
This system is used for disc formatting, data recovery, and such, but I use a LiveCD OS to run it.
i think that is a troll comment...
Ubuntu has no such thng as a safety net if you're used to run as admin with 'sudo bash' and then typing commands as normal.
In this case 'rm -rf *' WILL get executed even on latest LTS or non-LTS release of Ubuntu.
reason: on Ubuntu the root account is disabled by default (you must enable it manually), and the safety net alias for rm is not enabled when you become root with 'sudo bash'.
(That safety net is only enabled if you enable the account and then login normally as root)
In the past, to incentivise employee to pull all nighter to get that feature in before the deadline, was to pay them handsomely for it.
Now, to make employee come for all nighter is to pay them not enough for them to afford heating or even rent.
Punctual employee is the one who lives in their car, in company's car park.
Why does anyone think that an all-nighter is somehow a good thing, no matter what the pay is? It's pretty much guaranteed to leave you so tired you make stupid mistakes, just like Alejandro did.
If it's really that urgent (and it almost never is), put a team together and work shifts. Solo heroics are a recipe for disaster, and Alejandro's manager shouldn't have allowed him to do it.
Tell that to any bookkeeper, anywhere.
In addition, cutting staff reduces outgoings for which management gets rewards, whereas consequential failures are always blamed on the overworked staff left after that exercise.
If you want to fix that, find a way to rescind bonus payments when the consequences arrive. I wish you luck..
Spot on. In my youth, I got roped into trying to start a company, it was me and this other guy (who turned out to be ... nevermind). And yeah, to get stuff to customer then coding for 48 or even 72 hours in a row wasn't unknown. Debugging the crap that I had written while sleep-deprived took even longer.
The fact that there are duty time limits for pilots is instructive. The fact that these limits have to be enshrined in law so that airline execs don't push pilots too far is also instructive.
I was working on a COBOL based finance system in the late 1980s, running on a VAX. I booked out the code I needed to adjust (it was part of the preparations for the Y2K, turning 2-digit years into 4-digit years). I did some work and then had a brainwave...
So I stopped what I was doing copied the .cob files I was working on to .cobol, then went back to the .cob files and re-wrote the code. 2 days later, I had tested the code and the way I had made the changes was better than what I had originally started, so I deleted the .cob files... NO WAIT! GAH! I deleted the actual changes! I had to copy the .cobol back to .cob and start all over again. At least I knew what I had done the first time around, so it only took a few hours, not 2 days, to write the code again, in fact I came up with some other improvements along the way as well and the code was even more efficient...
Almost a parable for version control. ;)
Even venerable RCS (1980s) would have been a life saver here. I used it under MS-DOS in the 80s+.
I had a vague idea that VMS had versioned files as I recall something like purge name.ext;* but might have been another DEC OS.
We had version control, I had booked the code out, but, because I had started work on the code, but it wasn't compilable, because it was incomplete, when I had my brainwave, I couldn't book it back in, so I made a local backup, in case my idea didn't work and I had to go back to what I had started...
Sometimes I do this deliberately. Not exactly delete all the changes I made, just treat them as a learning experience and start again from scratch. Maybe keeping a header file of useful data structures. You can often redo 5 days of work from a clean sheet in just a day, with prior knowledge. You end up with better code than spending 6 days on something.
I once finished 6 months of work to develop a new internal app, without no spec and no clear guidance of what they wanted .... until I showed them something, and they could tell me what was wrong with it.
After it was rolled out I then told my boss that my next job should be writing version 2 ..... and I was totally gobsmacked when he said 'Yes'!
It took less than 6 weeks to reverse engineer a set of requirements out of version 1, then write a new set of code that was soooooo much better!
Been their, particularly with Access DB's. And as I have said before, never script a change to a SQL database with an Alter statement. Use the create instead. Then when you actually hit run through stupidity or an all nighter, it will whinge and tell you "object already exists' Only took me about 5 years of fuck ups to drill that into whatever thinks between my ears.
One fine day my task was to install WordPerfect onto a multi-user Xenix system which had about 30 simultaneous users.
I was introduced to the client org's Big Boss' assistant, Todd, and was left to get on with things. I asked, "Todd, will you please show me your most-recent backup tape?"
Todd (in a paranoid, demanding tone): "Why?! What's wrong?! What happened?!"
Me (in a reassuring tone): "Nothing's wrong. I haven't even started yet. I just want to see the tape and check it out. If I had said, 'Todd, will you please get out your most-recent backup tape?', then you'd know something had gone wrong."
With suspicion and reluctance, Todd went into a little closet, opened up a cardboard box of DC-600A tape cartridges, and handed me one. It looked new and dust-free; the label had been written on only once, with a pencilled notation of yesterday's date. I popped it into the tape drive of the Xenix server, logged in, su'd to root, rewound the tape, and did a tar -tv /dev/st0. The tape drive made noises, and after a few monents a list of directories/files flew up the screen. All good. I popped the tape and handed it back to Todd.
The WordPerfect package included a pile of 3.5" diskettes and a beautifully-designed and printed manual. I turned to the installation section and got started.
There was a long list of steps, and part of me wanted to tick them off with a pencil after I completed each one, but I could not bring myself to deface the customer's beautiful manual.
Soldiering on, I did this and that, then (per instructions) removed all the files from a temp directory via rm *, then (per instructions) did a cd .. And for good measure, did -- not in the instructions -- a pwd and an ls.
I was at the filesystem root, and there were no files. Subdirectories, yes, but no files. I yanked my hands away from the keyboard so fast I and my wheelie chair rolled backwards all the way across the room. I listened for the screams of angry users, but heard none. I sat there thinking carefully, and concluded:
(0) I had lost my place in the instructions, and had cd'd up to '/' from '/tmp' before I done the rm *;
(1) The files in the filesystem root are needed only for the boot process. Once the OS kernel is in-core (well, 'in-RAM'), those files aren't needed again until the next reboot (I glanced over at the UPS quietly humming on the floor);
(2) I should be able to restore just the files in '/' and everything will be good again; and,
(3) From here on, I will mark up the client's beautiful manuals (in pencil)!
I found the Big Boss' assistant, and asked, "Todd, will you please get out your most-recent backup tape?"
I was on the rota for out of hours customer support with a large computer manufacturer, in the days before good remote access to the systems.
The pager went off (I did say it was a while back), and I called the customer (a large car manufacturer) on the number they left with the call handlers. They said "We were doing some maintenance, and we inadvertently ran 'rm -fr .' in the top level of our database filesystem. Is there anything we can do to get the files back?".
My first question after hearing this (with a sinking heart) was "What did you do after realising what had been done". The reply "We spotted it quickly, and hit the power switch on the system, and decided to call you", to which I breathed a sigh of relief, and replied "Good call, there is a distinct possibility that we can get at least some of the files back. This is going to be quite complex, Don't touch the system until I call you again in 10 minutes after I've driven in to the office".
Once I had access to our test systems to test the procedure, I then went through telling them how to boot the system into maintenance mode, disable the automatic fsck of the file systems, followed by writing on the fly a script to do a istat on all of the inodes in the filesystem, finding all of the inodes that had a link count of 0 but still had a non-zero size listed in the inode. Identifying these, I wrote a little 10 line script that drove fsdb to set the link count to 1 in each of these inodes, which I delivered by fax to them directly from our systems, as I already had their fax number in our customer database. Thank goodness the process that cleaned the in inodes ran asynchronously with the command that unlinked the files!
Having done that, I told them to let fsck loose on the filesystem, and all of the files appeared in the lost+found directory, albeit without their proper filenames. I said that I could not do very much to restore the file and directory names, but they said that they thought they could work with it, because although the last backup was a little old, they did have a current listing of all of the files that were in the filesystem which included the inode numbers (I didn't ask why they had that but not a backup, at that point I didn't care too much).
As it was now about 4AM, I said to them that there was little more I could do and the rest was mostly up to them, but they could call me again if they needed something else. I didn't get a call again, and went back to bed, getting into the office a few hours late the next day, as per the callout rules for a dead-of-night callout.
I was curious about what the customer had managed to do, so I tracked down the person who had followed up the call in the morning. I was told that the customer managed to get all of the important database tables back, the only ones they couldn't were slow change data, and still fairly current on the last backup, so could be recovered them from that.
But they had told the person following up that they were very pleased with what had been done, and the call could be closed. So the person doing the follow up got the acknowledgement of a good and quick closing code on the call for their stats, and I did not get any of that against my stats! I was quite cross, at least for a short time.
Having begun my career on Unix back in the dumb terminal days, this was the one thing that we were taught straight away that we must never do without first doing a series of checks : pwd to see where you are, ls -al to check what's going to be deleted, then a double check to be absolutely certain that you were right with a second more experienced pair of eyes watching to make sure.
Way back in the midst of times when sh was all you got, along with maybe csh as possible shells, there was much rejoicing when tcsh was installed and you could edit the prompt to show various things, like the current directory you were in, along with the hostname you were currently logged into. This has saved lots and lots of time and gnashing of teeth over the years...
Now days bash is all you really need anymore, it's got all the features of tcsh and none of the downsides of actually scripting in (t)csh as well. Whew.
Having done a 24 hour walk in the Lakes a couple of years ago, and completely bombing during the circadian low, there are no apologies necessary for a FUBAR....
However, first job out of school was floor manager at Maccy D's. We had our bi-annual replacement of fluorescent tubes once.... The all nighter consisted of about 3 hours of work, followed by hours of playing star wars with the tubes in the attic.... This is the kind of all nighter we can all get behind :-)
I had to travel to the far east for an important overnight performance test.
I travelled economy arriving at 10:00 tired and jet lagged and was told to go straight to the customer!
I had asked if I could go to the hotel for a couple of hours sleep, but no... I had to go to the customer in case they wanted to ask me any questions.
I sat around, dozing for half an hour at a time. 10 pm we were ready for the big test. For my part the performance was poor, but could not see why.
As I was falling asleep on my feet, they let me doze on the comfortable chair. 3 am I woke up with inspiration...
I asked the computer equivalent of "did you release the handbrake?"... to which the answer came back "Oh dear,no".
We were lucky, there was a performance problem else where, and even with our handbrake on ... it was fast enough.
Within my area we had a debrief, and I asked, why the hand brake was left on... "we didn't think it was important"
I learned several lessons from this... and passed them on - such as always check, never assume.
I had an employer who didn't understand this. Travel cattle class for 10 hrs, across multiple timezones, and be expected to go straight from the airport to the customer site. Nope. Only in case of emergency, and even then, triple check everything you do, with customer sign off. And have good deodorant.
For some years, I worked for a government contract for the tending of Data General minicomputers. There were a lot of machines and a lot of operators and analysts, so that it was probably inevitable that now and then somebody would
CDIR :
DEL #
Once I was that somebody. What I discovered early on was that the peripheral manager, :PMNGR, was early in the usual order of files and so one of the first to be deleted. But once it was gone, there was no way for the system to delete (or create) files. Owing to our loose discipline in cleaning up after updates, there was usually a copy of :PMNGR.x.y.z that one could rename, and from there begin the recovery. And I think that the CLI used a sort of breadth-first discipline, meaning that the subdirectories of root were not purged until all the standard files were. One did not want to run with an obsolete :PMNGR, so that one then had to go to the latest systape.
Then there was the customer's employee at a previous job who was working his way through the commands and utilities book one evening until he got to FORMAT. It worked as described, but not as he'd have desired.
Is to be the only member of the team who did *not* work on the "to be installed tomorrow on the very expensive customer kit for first acceptance tests" (was working on the off-kit PC subsystem that (a) wasn't part of the test and (b) was in good enough shape to show the data from the kit was being collected, it just didn't have all the post-processing in place).
But, the *entire* team was going to stay overnight.
Oh, I also lived further away than anyone else, so next morning, I drove out of the company carpark, pulled over next an undeveloped field (peaceful) and slept - not napped - before going on to the motorway home.
I was doing a shopping cart upgrade.
Got all the things in place, extracted files here, test things out, now I just need to delete the archive from the root of the web server folder, and done.
I click the file. Alt-tab around, check some things, double, triple check. Alt-tab back to Windows Explorer. See the file highlighted in blue. Shift-delete-enter. Deleting file 169 of 4096..... *WHAT?!?!?!?*
Windows (7) being Windows, Windows Explorer got rid of the "Change the highlight color of the selection to gray when focus shifts away from that UI element." So despite the file being highlighted blue (active color), the focus was now on the folder tree. I couldn't see that, and 11:05pm upgrade (not before my bedtime, but people talk about evening work ;-) -- it just... all looked correct. Shift-delete, and in one all-encompassing set of keystrokes, enter.
I ended up deleting the shopping cart for an online business for which a brief interruption in sales was a *serious* event. (Most of those sales occurred during the day, so I was allowed to take the cart to maintenance mode in the evening - these were not impulse purchases, but to be down for a day would be ....)
---
It turned out that due to one of my nervous ticks, I "right-click -> send-to -> zip file" the shopping cart folder before beginning the update procedure. So I deleted the shopping cart, caused a complete outage, but recovered in under three minutes. :-) LUCKILY the software didn't depend on files having specific inodes, or permissions, or weird things like that. Luckily the IIS app pool* just picked things back up.
"What about your backups?!" well, for systems created by hand, in an 8-employee shop, (-: the backup was probably 8 months or so old.
---
* IIS app pool. "Can you figure out why our customers lose their shopping carts when they leave their browser open?" This of course came with just the above information. Later we identified this as happening over-night. Sometimes, not always. Later still, I found out that IIS actually restarts its application pool every 16 hours to prevent memory leaks and alleviate application errors -- Windows needs rebooted every once in a while, see, and so, too, probably, do IIS applications. So IIS reboots for you. When that happens, the session IDs are invalidated, and... the active cookies are worthless, and the shopping cart goes away. You can increase this timeout, or I think disable the timer'd restarts of your app pool, but omfg windows.
> * IIS app pool
Dave Cutler, one of the geniuses at MS, speaks how microsoft.com webseite was the first real usage of Windows AMD-x64 due to those memory leaks, just right after their first "hey, it boots on real hardware without crash!" built. Listen to the full thing of how he talks about Windows XP before the first service pack, and his altername name for it. We currently have a "history repeating" thing about that...
I think he an his team mus rescue Windows again...
A long time ago in the mid 1990's I was the sole unix and oracle admin for a "medium sized company" that run its ERP system on an HP-UX box. As with these systems space began to run out and so more disks were added to increase the space.
For some reason I can't remember I had to destroy and recreate a filesystem, and finding space short to move the files around I thought OK, I'll ftp them off to my windows desktop machine as they were a bunch of small but numerous files.
All worked out fine until I fired the system up at the end and found out that at the windows end the FTP client had converted all the file names into upper case and the system could not find them any more. after trying to rename them on the fly as the errors cropped up and gave up and found out that yes the backups had worked and yes they could be restored.
But the cold sweat feeling until the systems fired up OK was not nice, and I was watching the error logs all the next week in case I had missed anything!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
