Final step to put new website into production deleted it instead

Welcome to Monday! The Register hopes you arrive at your desk well-rested after a pleasant weekend, and not stressed out by working late as is the case in this week's instalment of "Who, Me?" – the reader contributed column that chronicles your mistakes and escapes. This week, meet a reader we'll Regomize as "Tom" who in 2009 …

[Korev]

Tom Asda be lucky it came up again so quickly

[KittenHuffer]

He should Tesco and get some sleep!

[KittenHuffer]

After working Aldi I'm not surprised he was too tired to stop the mistake.

[KittenHuffer]

It was only a Lidl mistake but it had huge consequences!

[KittenHuffer]

This wasn't a mistake ..... it was an M&S mistake!

[Ozumo]

There should have been a Safeway to do this.

[blu3b3rry]

The in-house tech should have been more co-operative

[Korev]

I'm pleased Tom Waitrose to the occasion

[wolfetone]

Definitely, such a Kwiksave

[Doctor Syntax]

Morrisons to double check before you hit return.

[Doctor Syntax]

Always Sainsbury your mistakes otherwise you might get sent to Iceland.

[wolfetone]

Don't Budgen on that.

[Korev]

Never Onestop the puns

[handle handle]

I don't Carrefour all these puns.

[Benegesserict Cumbersomberbatch]

Puns are Coles to Newcastle around here

[Benegesserict Cumbersomberbatch]

A reminder to Pack & Save your work (especially in NZ).

[Scott 53]

It's a Nisa outcome than I expected

[collinsl]

People really should Onestop Spar-ring with one another about this, or take it to Somerfield

[katrinab]

Safeway was taken over by Morrisons in 2005, so it couldn't be them.

It also wouldn't have been Morrisons because they didn't sell food online at that point, they were very late to the game.

https://www.theguardian.com/business/2011/feb/15/morrisons-buys-kiddicare-seventy-million

[PRR]

I shopped Safeway in 1959 in California. (Well, my mother shopped, I just threw Sugar Pops in the cart.)

> Safeway was taken over by Morrisons in 2005, so it couldn't be them.

About six megamarts own all the supermarkets in the US under many store-names.

Safeway 2026 has a perfectly good and deep site https://www.safeway.com/

It is now a bud of Albertsons, but DBA Safeway.

What I found mildly amusing is: I'm now 66 years and 2,800 miles away, but I ask Safeway.com for my nearest store, it gives me a store 7 miles up the road from me. That one says Shaws on front, but I know the store-generic products are the same badging "Signature Select". In some sense I have shopped there all my life.

[IvyKing]

In SoCal the Albertsons stores use the Vons name and the Kroger stores use the Ralphs name. I'm not sure how Lucky we are to have only a few grocery store chains still around.

FWIW, Lucky was the name of a grocery store chain common in SoCal a30 to 40 years ago.

[PRR]

> In SoCal ........ the Ralphs name.

Are you old enuff to remember when they hung an effigy of "Ralph" in the store, with a "High Prices" sign on his chest? This would be pre-1960.

[Sparkypatrick]

Safeway Inc opened stores in the UK in the 60s. In the 80s they sold the UK business to the Argyll Group, who owned Presto. The Presto stores were rebranded as Safeway or sold off to Spar. The chain struggled and was eventually bought out by Morrisons - a Northern England brand that wanted to expand South.*

*Not exclusively South. I have shopped in a Safeway in Kirkwall (Orkney). Can't get much further North in the UK, unless they had a branch in Shetland.

[Sparkypatrick]

Assuming it's a UK chain, Tesco. Their website notably combines online groceries and a range of other household goods (often from third parties) delivered separately.

[Uplink]

An M&S-take?

[Efer Brick]

he did the nisassary

[Aladdin Sane]

The employee really should've found a Safeway to do it.

[Korev]

I'm afraid someone has already Somerfielded Safeway...

[Korev]

That's Spar for the course these days...

[Aladdin Sane]

Argyll and get my coat then.

[Anonymous Coward]

"when it's gone it's gone"!

[Michael H.F. Wilkinson]

Ah, the old "rm -rf *" command

Very much the nuclear option (hence the icon). I always triple check whether I am logged into the right machine, and in the right directory, and have a back-up of said directory, before hitting the nuke button. And even then I feel nervous about pressing it.

[Anonymous Coward]

Re: Ah, the old "rm -rf *" command

Had a developer screw a system up by writing a script that used a parameter something like:

rm -rf $MYPATH/*

Luckily he wasn't root, but it caused enough problems that it took a while to get everything back up and running.

Only for him to tell me "I only did this" and proceeded to DO IT AGAIN.

I've also seen people do the same thing with a chmod, but as root.

[Dan 55]

Re: Ah, the old "rm -rf *" command

Had a developer screw a system up by writing a script that used a parameter something like:

rm -rf $MYPATH/*

set -u at the top of the script. Always.

[Joe Gurman]

Re: Ah, the old "rm -rf *" command

Did you beat the manslaughter rap, or are you writing from one of HM prisons?

[Anonymous Coward]

Re: Ah, the old "rm -rf *" command

Only just got out.

[vulture65537]

Re: Ah, the old "rm -rf *" command

I knew people who let an unset variable prevent them from deleting user accounts for years. They had a script hooked into the user deletion work and without a counter it had an infinite loop.

[Will Godfrey]

Re: Ah, the old "rm -rf *" command

Actually I can't remember ever having used it. Early on I received dire warnings about my life expectancy from an experienced dev.

[Strahd Ivarius]

Re: Ah, the old "rm -rf *" command

if you had ever used it, you wouldn't have been in a position to remember anything...

[werdsmith]

Re: Ah, the old "rm -rf *" command

Just never do it ever, it’s a taboo command under any circumstances. Like calling my mum a c***

[David Hicklin]

Re: Ah, the old "rm -rf *" command

But fun when decommissioning and seeing how long the system will stay up

[Robert Carnegie]

Re: Ah, the old "rm -rf *" command

If the system you run it on is the system that you're decommissioning. How sure are you?

[Evil Auditor]

Re: Ah, the old "rm -rf *" command

...and even after the same triple check, I did get it wrong. (Those bloody machine names, they all look the same to me...)

[Cloudseer]

Rename the directory, we are going to need it later. To delete it.

[simonlb]

Provided it isn't linked somewhere, although that can be addressed. Otherwise, definitely best practice. Have one of these --->

[KittenHuffer]

The secret to intelligent tinkering .... is to keep all the pieces!

[Korev]

Ahhhh

website_backup.tar.gz

website_backup_old.tar.gz

website-backup_old2.tar.gz

website_backup4.tar.bz

website_backup4_dave.tar.bz

[blu3b3rry]

You forgot one.

aaa_111_website_backup_new.tar.gz

[wolfetone]

And these:

new_website_backup.tar.gz

new_website_backup2.tar.gz

[Anonymous Coward]

new_website_backup_final.tar.gz

new_website_backup_final2.tar.gz

[Cris E]

new_website_backup_final2.april.tar.gz

new_website_backup_final2.april2021.tar.gz

new_website_backup_final2.april2023.tar.gz

new_website_backup_final2.april2027.tar.gz

[Evil Auditor]

And

new_website_backup_old.tar.gz

old_website_backup_new2.tar.gz

[Paul Hovnanian]

Of course, you are saving all these backups in different locations in the file system. Or even different file systems.

Or you'll still lose them all following the inevitable rm -rf *

[Bebu sa Ware]

Rename the directory, we are going to need it later. To delete it.

tar - cpSf /somewhere/safe/precious.tar.gz ./precious

mv ./precious ./precious.bak

My motto: "Make your mistakes slowly, keep them small and easily reversible, if possible."

Applies equally to powertools which is why I eschew them for their manual predecessors.

Qui festinat res destruit.

[Caver_Dave]

Re: Rename the directory, we are going to need it later. To delete it.

"Festina lente" make haste slowly

[An_Old_Dog]

dd

dd if=/dev/zero of=/dev/sda bs=16384 [Enter]

Time passes ... "Hmmm .... that's taking a long time for such a low-capacity flash drive." ... I idlely noted the furious blinking of an access LED inside the hard drive bay ... inside the hard drive bay!

^C^C^C^C^C

Too late. I did not have a backup of the now-zeroed boot/OS drive.

Fortunately, I had, elsewhere, a copy of the response file and scripts I used to install the OS.

Afterward, I built a PC with dual DVD burners, and some removable drive bays, just for such operations. The OS ran from a DVD, to minimize the consequences of such future mistakes.

[Korev]

Re: dd

When I used to name old HDDs, I always used to unplug the new discs and boot from a CD

[Korev]

Re: dd

> When I used to name old HDDs, I always used to unplug the new discs and boot from a CD

Nuke, not name. At least I got the icon in the post right...

[Doctor Syntax]

Re: dd

When ddi-ing an image onto a USB drive there's always a heart-stopping second or ten (at least that;s what it feels like) between pressing return and the LED on the drive starting to blink.

[DS999]

Re: dd

When you run Linux as the logged in user removable drives that are automatically mounted are writable by your userid but system drives are not.

You can get the same behavior on servers you're logged in remotely to by using a mount command that specifies a particular uid and/or gid, allowing writes to that drive to happen as a normal user instead of root. Then you can execute commands to write to the removable (or newly installed) drive as a normal user, and don't have to double/triple check every dd command because it isn't possible for you to accidentally overwrite a system drive.

[Autonomous Mallard]

Re: dd

I always use the /dev/disk/by-id or by-uuid entries these days, haven't blown away the wrong disk in a long while now. In /etc/fstab as well, since the /dev/sdX order is non-deterministic between boots.

YMMV based on udev configuration: https://wiki.debian.org/udev

[Sam not the Viking]

A very angry customer phoned our MD to say our software had caused a major incident; machinery had failed to operate and the consequences were serious. Not life-threatening but the local MP was involved. We were summoned immediately to site to meet with the end-user, a representative from the EA, the consultant, the main contractor and the leader of the Parish Council..... As the person considered best to offer as a sacrifice, (longest in the tooth), I was sent to investigate and download our logging/monitoring data. Tempers had eased by the time I got there but the tension was palpable.

Our logging device was simple but the data was not easily retrieved. Once in the software, the keystrokes to save or delete were easily confused...... as from experience, I knew...... Under watchful eyes, I carefully downloaded the machine history under the watchful eye of the consultant and gave him a copy of the unabridged data.

It was blindingly obvious what had happened; the machines had not been switched to 'Auto' and so were effectively switched 'Off'. Cue huge embarrassment to all (others) involved. We thought it prudent not to invoice for our call-out.

I still worry what would have happened if I had...... No I don't want to think about it ----->

[Lee D]

Delete

Never delete.

There should be absolutely no need to delete. Ever.

You rename, copy it somewhere else, or you move it (being careful that you're not moving it over the top of something else).

You can even move it off to backup / temporary storage.

But there's no need to delete.

No need to delete emails. No need to delete anything running in production. Ever.

The only place you ever delete from is your years-old backups of things that aren't even used any more because you've been moving them further and further and further from your production systems and live backups until by then you're certain they will never be needed, and you've already copied them to some archival-type backup. Those are the only things you ever delete. Things you haven't touched in 10 years.

Honestly, I think the delete command should just be removed from users. We kind of did this with Recycle Bin, etc. but there's no need for them to actually delete anything on a managed system. And there's no need for me, the person managing that system, to delete anything that's in production, ever.

You just keep moving it around until it's clear that it's NOT used by anything (because it would have broken a dozen times already by then), then you do a final move off the system.

You don't ever need to delete.

Definitely not with a wildcard. Definitely not with a -f. Definitely not with some dumb tool to execute the same command across dozens of servers.

And if your production system and its various storages, etc. "doesn't have room" for something... well, that's a problem in itself if it ever comes to restoring those systems because you just don't have the elbow-room to manoevure and verify as you go.

Deleting is setting things on fire. Moving them is putting them on the side, then putting them in the loft, then taking them out of the loft and putting them in storage, then putting that stuff that's been in storage for years untouched into a bin, then putting that bin into the outside bin, then actually letting the bin be taken away.

Honestly, so many people who administer systems have such a blasé attitude to actually handling data, it's taken me years to drum it into those people who work with/under me.

"I'll just delete..."

"No you won't. You'll rename it and move it out of the way."

"WHY!?!"

<twenty minutes later>

"Do you see why now? Now just copy back what you had there originally and do it again. And COPY it back. Keep that clean copy clean and out of the way of what you're doing."

"But don't we have backups?"

"Yes. And I hope never to ever use them, and you should have zero reliance on them for what you're doing. Even before you started this, *I* copied that folder somewhere where you can't see, in case it went wrong, purely because I don't ever want to have to restore from an official backup."

[KittenHuffer]

Re: Delete

TLDD - (Too Long, Don't Delete!)

[Doctor Syntax]

Re: Delete

Move is not safe. A customer had an old file tree under root that we wanted to move out of the way.

mv / $destination

Oops. Caught it too late. Just as bad as rm if it's moved so much that only your shell is working. This was SCO and it had been built with special driver for the multi-serial board. It took hours to get the driver emailed from the vendor.

[Autonomous Mallard]

Re: Delete

I try to avoid absolute paths entirely in mv, rm, etc; e.g:

cd /srv

mv website website.old

It at least limits the blast radius somewhat. I have shot myself in the foot before with mv's default behavior of overwriting anything with the same name in the destination folder, though.

[mtp]

Re: Delete

I have a directory called FromOldPc, in this is a subdirectory called FromOldPc, ... at least 5 layers deep by now

[J.G.Harston]

Re: Delete

Gawd, I'm so boring.

201407 201506 201606 201611 201708 201709 201712 201806 etc.

[M.V. Lipvig]

Re: Delete

I do something similar, and have stuff from 30 years ago buried in there somewhere. The only real negative to this is it means there are multiple copies of the same files in there. Not a really big deal though, since storage size goes up a lot more between computers than I can generate.

I also save everything to a file on the top desktop. The computer is not allowed to save anything to normal default locations. The advantage, if I need to back the data up I just stuff in the portable storage of the moment, then copy that one directory to the portable storage. It takes me about 15 seconds to start the process, 30 if I need to format storage first.

[Steven Kudelko]

Re: Delete

Every so often I just get overwhelmed with the amount of crap accumulated in my ~/Downloads and ~/Desktop folders and just move everything to my file server.

I probably have at least 10 folders on the root of one of the drives named variations of "__MacBook" "__from Lenovo" "_mom's iMac" etc. with the number of underscores at the beginning representing how important I thought it would be to go through these backups and extract what I'd need to keep.

I think the only thing I've ever needed to extract was a PDF scan of my photo ID and that was because I couldn't be bothered to just do a new one.

[El.Mich.]

Re: Delete

Wow!!! :-)

Thanks a lot for this summary of doing IT correctly! :-) [And probably just the old-fashioned way]

On this occasion: I probably should go and check once again if my oldest some 500-MB-[sic!] -drives are still in good working order! ;-) Because they probably still contain some data that has been moved ... since the late Pentium-II-era ...

Cheers to you! :-)

At least no one who has ever read your post will be allowed to ever say again: " ... but nobody told me that! And I just wanted to save some space ..."

THANKS a lot!!! :-)

Unfortunately I can only upvote your post once! :-(

[I could be a dog really]

Re: Delete

You don't ever need to delete.

Hmm, how many ways is that wrong, lets think of a few ...

1) It's illegal to keep personal information that's no longer needed.

2) disk space isn't infinite, but the ability to use it is. For example, I tend to rip DVDs and store just the film/programs on my media server with the physical disks put out of the way - it's both more convenient, and a better experience stripping all the "coming soon trailers for stuff that came out decades ago, and the obligatory 'you're a criminal' bits". That takes "a few gigs" each time with stuff I will never need again (and if I did, it's on the original DVD). I only have a 2TB SSD in my laptop, spinning rust (if I could afford to waste it like that) would be a lot slower over USB.

3) In a past life, we had an accounting system I was sysadmin for - it generated gigs of data when 18G was about the limit if you had deep pockets. We often made copies of the database in order to test something. Ditto it not being practical to keep everything.

4, 5, 6, 7,...) Some of us have discovered something called ... a backup. Used to have cabinets full of the tapes for said system in #3

Definitely not with a wildcard. Definitely not with a -f.

Hmm, so are you going to delete hundreds, thousands, of files individually ? Ditto hitting Y "a few times" when the OS seems to think it's fun to keep asking ?

The general principle - move something rather than delete it - yes, fully agree. But unless you have an unlimited budget for storage and are willing to accept the risks of constantly upgrading the hardware, you have to delete stuff sometime when you're done with it.

[disgruntled yank]

Re: Delete

.> No need to delete emails.

Talk to the folks in legal. There is much to be said for having policy, set and enforced long before trouble comes, saying that all emails shall be purged after a certain period.

[Scott 26]

Re: Delete

I was doing a decom of a "small yet highly public business group stood up for a public inquiry" for a govt.nz... one of the 'consultants' kept EVERY SINGLE EMAIL.... including "Coffee in cafe XYZ at 10?' style emails. But worse: he also saved the .msg/.eml files for EVERY SINGLE EMAIL into his Home drive.

There's "everything we do is subject to the Public Records Act" and then there's this guy.

[Yes Me]

Re: Delete

"No need to delete emails. "

Um, with all due respect (i.e., none), there is sometimes a compelling need to delete emails.

Recent partial and redacted disclosures by the US DoJ offer thousands of examples of failures to delete.

[phuzz]

Re: Delete

I used to have a boss (Hi Malc!) who would delete pretty much every email that came into his inbox, unless it needed to be worked on right away. Then he'd delete it.

He used the 'Bin' folder the way most of us would use our inbox.

This actually worked for him, until the day he'd got me to do some other work to his email client while he was busy in a meeting, and I decided to be helpful and emptied his deleted items. He was annoyed, but had to admit that his ridiculous filing system was the real problem.

[I could be a dog really]

Re: Delete

Back in the days of the Mac Classic I think it was, we had a client who did the same with files - she wasn't happy when the wastebasket got emptied. Icon suggests the fate of the files, and her response.

[Tim99]

Re: Delete

Generally, I avoid deleting. My current mail system has a folder called 'ArchivedSent' containing emails going back to March 2005. The performance is still good - searching for a person's first name found 272 messages, several going back to 2005, and took about 1 second. In all there are just over 41,500 messages in 31 local folders, occupying a total of 4.43 GB - No, I don't use MS Exchange or Outlook...

[Apocalypso - a cheery end to the world]

The thing that stood out for me...

The thing that stood out for me (aside from removing 6000 lines of superfluous script - someone was being paid by the line!) was:

"We had carried out multiple dry runs, deployed and rolled back in pre-production a number of times. And we had a four-hour window from 2:00 AM to 6:00 AM to when the business would allow the site to be down for this process."

Tom sat next to the employee who was allowed to make the change.

"I had supplied all the steps in detail, and all he really needed to do was cut and paste a few commands," he wrote.

If you're not allowed to make the change yourself then the person who is must do the rehearsals. Otherwise the rehearsal is not a rehearsal. If the boss complains just ask him to imagine rehearsing a play without the real actors being present.

<icon: Hamlet>

[TimMaher]

Re: “Hamlet”

rm or not rm? That is the question.

Whether it is better in the mind to suffer the slings and arrows of outrageous backups,

or, by deleting, end them?

[Herring`]

Re: The thing that stood out for me...

If you're not allowed to make the change yourself then the person who is must do the rehearsals.

Ah, but they are the special ones anointed by management. These gods are far too busy to participate in your rehearsals.

Seriously though, I do feel the pain. You bother to do a carefully tested list of steps that need to be carried out in order. Then when it goes tits, they come for you saying "it didn't work". And you ask

"Did you follow the procedure exactly?"

"Of course"

"Really?"

"Yes!"

"Every step, in this order?"

...

[cd]

I would have walked out after the first failure to follow my instructions.

[I am David Jones]

…leaving a sheet of paper setting out your new hourly rates, effective immediately.

[Scott 26]

This...

"why didn't follow my carefully wirtten and tested instructions?"

You deviate; you are on your own - Byeeeeee

[J.G.Harston]

Back in the day of CONFIG.SYS sometimes they were binary files. Also, the usual text editors would strip out anything they didn't see as text when loading.

One day a colleague loaded CONFIG.SYS into Notepad. "Ooo, it's all weird blobs." Yes, I told him. It's a binary file, you can't edit it, just quit.

Quit -> Do you want to save? -> Yes

Me: NOOOOOooooooo.......!!!!

Them: It's ok, I didn't change anything.

ARGH!!!!! Yes, *YOU* didn't change anything, but Notepad has changed it, stripping everything out that wasn't a text character.

[Outski]

Using Notepad - first mistake

[Boris the Cockroach]

32 metres a minute

is not the speed to have an 'oh no' second

Because thats the speed our machines go, which means about 400-500mm per second, so pressing the button (or enter) and the human reaction time for the oh no second is... well a second, followed by the 0.25 of second that the motor pathways to your arm muscles are fired before your body starts moving, followed by that long drawn out 0.5 of a second before your bash the e.stop and everything comes to a halt, then you find out your expensive machinery died about a second ago when the turret slammed into a rotating plate doing 6000 rpm and is now very firmly welded to it.

On the bright side, you'll have some stories to tell, as will the machinery techs who prise the remains apart.

On the bad side , your first appointment at the job center is 9am, the bus is late and its raining.

[SuperGeek]

Oops!

What a Costcutter measure!

[Raphael]

Many moons ago I was onsite with a client (a NSW council that shall remain nameless) doing a production deploy of a new tool that integrated with their document management system.

It was going well. And although I wasn't allowed to actually touch a keyboard when things were being done on Prod there were no hiccups, each step being documented as we went.....until the end, we were actually finished, when someone else accidentally restored a backup meant for another environment over our work for the morning.

I had to catch a taxi in 45 minutes to get to the airport in time for my flight back to New Zealand, so was thrust into the chair and asked how quickly I could redo all the steps on prod we had painstakingly done in the morning, as it needed to be signed off as complete before I left.

I did it with 15 minutes to spare to run down to the taxi rank (10 minutes run away) and just managed to make my flight home