Never put off until tomorrow what someone could erase today Greetings once again, gentle reader, and welcome to another instalment of Who, Me? in which Reg readers like yourselves soften the start of the work week with reminders that we all sometimes make mistakes. This week's mere mortal is someone we'll Regomize as "Christopher" who worked, back in the halcyon days of the 1990s, as a …
Actually there is nothing wrong with that, as long as you don't have to plug it into the server directly, though I bet on you had to. If you needed more than one for one backup run: On the client station combine several of them as RAID0 or Storage-Space stripset with the number of column = number of drives. Increases the throughput, more than you think since the access times are evenly spread across the drives too. I know cause I combine a few shingled-magnetic USB drives that way for my private backup, and the speedup was more than expected.
But that is not the professional company way :D:
Oh I don't know.
On the face of it, I'd agree - but on the flip side you are relying on other to tell you whether or not it's needed. I have done transformations/migrations (from ancient HP-UX to Linux), left the old systems around for 6 months (crappy decommissioning process), only to be asked 6 months later by a developer if we could "copy over some stuff he'd forgotten about at the time". That's despite a multitude of reminders, change requests, and the system being powered off for some time before removal.
Backing something up isn't particularly helpful if you have nowhere to restore it to....
Years ago I did a 'simple' file server migration - windows 2000 to windows 2003. (or it might have been NT to 2008).
But I took the opportunity to scan the shares and build a histogram of file count vs age, so we could go to the business with the analysis "do you really need this share?" and have a bit of a tidy up.
One share had files with the newest date of 95 months ago.... so I went to the business with my spiel "hey these haven't been touched in 95 months.... etc"
The business came back "95 months....???? Oh, that's 8 years. Oh yeah, that share was for a project that shut down 8 years ago, so yeah that'll be about right"
me: "oh cool - dont need it any more then"
business: "ummmmmm...... if we did, could it be restored and how long would it take "
me, internally: "FFS!!!!!"
With a full back-up on tape and an external tape drive to re-attach to, say, the developer's workstation if not a handy data-scratchpad for your own use, there'd be hassle but no problem.
As an administrator you know full well that someone will pop in for something, so you keep everything* unless told otherwise by law. But on tape or holographic-magneto-optical disc or whatever, not on-line. With a suitable drive with suitable attachment for current machines.
Also to Scott 26: Give a good workable answer and you get to archive more. There's more to it, but I'm sure you'll get it if you think of doing, or try to do, an actual functional restore. Which you should, this being backing up and all that.
* Well, the change-y bits, like /etc and /var and /opt and home directories, if any, and so on. Probably not the basic stuff from the install tarball, since you have that already and know^Wwrote down what was in use for that particular server.
Or Zummerzet.....
Rainy Friday Afternoon post lunch (No beer), some deskside\level 2 or 3 bods wheel down some kit for wiping & disposal.
This was done over the Friday afternoon\weekend.
E-waste disposal place picked it up with the rest of the kit for disposal sometime the following week.
The following Friday (Also raining) post lunch (No beer) phonecalls e-mails & even a physical presence in the IT Service Centre (Or a wooden hut attached to County Hall Offices & sited in the carpark) in increasing levels of panic.
"Do you have this server, the asset number?"
Back through our records...... "No!"
"Do you know where it is?"
"No!...........but we do know where it went" (Throwing them a bone).
"Where?"
"Somewhere north of Bristol, or at least it was!"
"WTF! ....WHY!"
"Your team members brought it down for wiping & disposal, this was done & it was taken off site 10 or so days ago!"
"FUCK!"
"Whats the urgency?" Say we, thinking it's about time we got to ask a question.
"It had bespoke "accounting\financial" software on it, all ready & configured for a consultant who is arriving on Monday, we left it powered down for (6) months & some PFY decided as it was doing nothing, it should be disposed off"
"Ahhhh well that's rather unfortunate!"" /SMIRK
"If you give us the contact number of the disposal company we can see about getting (Buying) it back"
"Sure.....if they still have it in one piece & not destroyed or broken down for spares or sold it onward to a buyer"
"WHAT!"
"We (The Council) paid them to take it away, it's theirs now, they can do what they want with it or do you think they should hold on to it on the off chance we might want it back in a hurry for 3 months".
They got hold of the company, bought it back at a very high price & collected it that afternoon (Place was closed weekends) after a high speed drive up & down the M5, then spent the weekend reinstalling it & restoring the configuration from backups. Just in time for the very very expensive consultants arrival Monday.
I was fortunate enough to use it when it was not only reasonably affordable but a single tape would hold a full backup of our entire storage or a week of incrementals. For a brief period it was almost perfect.
Before that it was Zip drives, which may partly explain my DAT-nostalgia.
I worked in IT and we were in the process of updating the OS on machines from NT 4 to XP. We asked everyone to ensure that they backed up ALL their data before we did the deed.
We were scheduled to upgrade the machines in the security office. They all said "Yes, we've backed up our data" including the lady who used a ZIP drive for such a purpose. The deed was done and people copied their data back. All except the ZIP drive user - all her disks were blank.
We checked all the disks and she wasn't lying (never trust a user when they say something isn't working). Due to this we asked her to demonstrate her back up process (which she supposedly did every day). The process involved putting the ZIP disk into the internal drive, waiting for the light to stop flashing, ejecting the disk, and putting the disk into a fireproof safe. No software was run, no files copied manually, nothing. Yes, dear reader, she thought that the drive checking a disk was present was the backup process.
Her colleagues quickly educated her as to how backups should be performed. They also watched her like a hawk. I have no idea if they ever got the data back from somewhere else or if it needed to be re-entered manually (details on 500+ employees)
Is this a different click of death? The one I think I remember was conventional hard drive heads, maybe specific to a generation of WD HDDs; I don't remember one associated with ZIP drives, as I think you're saying here, although I never used ZIP myself, only the later Jazz drives.
ZIP drives had it too, the drive heads would drift out of alignment over time, until the drive wrongly thought a disk was blank.
The worst part was, it was contagious. If you inserted such a corrupted disk into another drive, it too caught "Click of Death".
No backup is complete until you have demonstrated successful recovery. I would not expect average users to have complete confidence in Murphy's law but anyone working in IT for a month should have that expectation beaten into them by experience.
"Her colleagues quickly educated her as to how backups should be performed."
So, either she was never shown how, didn't learn how when shown, or blagged that she did know when asked if she knew. Still a process error thoiugh, she should have been shown and then had to demonstrate she could. The old, it's not a backup until you've restored" notwithstanding, but that's a different process that was also clearly flawed :-)
My recollections of DDS tapes (DAT was the term for the audio variant) are not very good.
While it was possible to write and read tapes on a single drive, I found drive inter-operability a bit of a problem. Just because you could read the tape back on the drive that created it, it did not mean that the tape would be readable on another drive.
I worked alongside the IBM AIX Hardware centre in the '90s, and the 4mm DDS drives that IBM shipped (which I'm sure were re-badged HP drives) were one of the most reported pieces of failed hardware that that group dealt with. The fix was often to re-align the head, but this required a diagnostic and calibration tape that had to be purchased from HP, and run through with diagnostic software that did not run on AIX or IBMi.
I preferred to not rely on them at all. We used Exabyte 8mm tape drives for critical backups, but even later models of this became problematic, particularly their need to be very regularly cleaned.
Since then, I've had good success with Quantum DLTs, IBM Magstar 3570, and the various generations of LTO tape. Where I'm currently woking, we've just put in some LTO9 drives, and these are phenomenal in their capacity and speed.
When I read that line in the article, I turned to SWMBO (I had been reading it aloud to her) and said, "what do you mean, remember? We're still using them at work!"
More precisely, DDS3 on the "ancient HP-UX" systems another commenter mentioned.
About 40 years ago I had to go on site in the US to assist with an upgrade to a city's computer. This was in the days before backup computers we used. It meant shutting the system down, and installing in place - which took an hour or so.
At 2AM the system was shutdown and the upgrade done. We went home happy.
Next morning there was a report saying that the police had stopped someone and wanted to check the FBI computer to see if the person was wanted. Because this occurred during our down time, the police could not get through, and the man was released.
The next morning we heard about this, and found the man was wanted for several murders and other crimes - and the police had let him go,
I learned about the need for available systems from this incident.
I'm a recovering CommVault user (not really, still have to use it) but having done multiple migrations of backup systems over the years, you *never* get rid of the old one or it's name until it's long past the "need to restore" date. In this case, going from DAT to DLT, along with new backup software would have required *zero* need to keep the backup server name the same.
In fact, I would have set them up in parallel and run them in parallel for a bit, so I can make sure things are working. So there's some serious incompetence here in terms of management, along with the techie.
Especially since CommVault is it's own special brand of special where you probably need a windows server, plus unix media agents to actually pull the data off things.
So I call BS on this story...
It's a story... embellished a bit... and from the freakin 90's when I started in this racket, and best practices were best effort.
Unless this software used IP and HOSTNAME for some devious reason, I follow your reasoning. I remember installing some bit of software that required using IP and HOSTNAME in the licensing setup, and then subsequent client connections needed both as well to connect to the software instance.
Previous employer had the SOP to simply hold on to the old hardware as-is, in perpetuity. Literally stacks of laptops from employees that departed years ago, just in case some legal issue came back to bite them. Some who were barely remembered because of a short tenure, the laptops practically new but now outdated.
A closed office, RIF'd IT staff, instructions to pay for e-waste services, and much of that kit was re-purposed out the side door. E-waste company wanted $50 per device, so we saved the company money. No one was left around who had passwords to any of the systems, so everything had to be wiped anyway.
Cheers to days of crappy inventory tracking!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
