BOFH: The Great Backup BACKDOWN "So I see that backup check went well?" the Boss asks, trying to drag me into a conversation that will have nothing to do with me, but sounds technical enough that I would be an idiot and say... "What backup check?" the PFY pre-empts me. NGAAAAAAARGH! "The backup check," says the Boss. "The Financial Director wanted to be …
Sadly, that is the story of our desktop support groups life.
We have a product that will backup specific data from a systems HDD (Word, Excel, and a few others...) BUT the system needs to be on the network for more that 5 minutes at a time.
We have a group of travelling types that are rarely in an office and it's even rarer for them to be on VPN any length of time.
Needless to say, their shit don't make it in to the storage vault.
Give us time. Assuming that you're on M$ we'll disable the write functionality to memory sticks with a GPO.
Then we'll wait a month or so for you to work out how to circumvent the GPO via the registry, spread the knowledge round your colleagues etc before blocking it again using an obscure option in the AV software.
Finally we will agree that some writing to USB is required, but not before agreeing to an insanely complicated exception procedure that expires at a random point in the future.
Before setting a rule that only encrypted USB sticks maybe used. We'll then specify an out of production model as the only one allowed.
Anon because I'm not making this up!
I remember disabling USB storage devices, I had to hide for weeks!
Weeks after several people brought in infected drives and they had to hide from the IT department (Its no wonder they outsourced us and changed our name).
Not Anon because I can't be bothered to click the Anon button
RE AC with the encrypted USB pens etc. I've seen this happen too. I strongly suggested against using pre-made encrypted USB pens in the event that a hole be found in their security, and instead suggested using an industry standard software suite for encrypting/decryping (the sort you can run off an unencrypted part of the drive to open it up) as if a hole is discovered, it can be patched.
Suffice to say when those devices were found to be worthless thanks to a glaring security hole (non-random random number generation or something that made decryption trivial - I forget the exact details) I said, in front of the 'wrong people' (IE the management who overruled me) "I told you so" and sloped off for a smoke.
You'd remember me if this were the same place; trenchcoat, scottish, grumpy, sarcastic...but I suspect this sort of shit happens *everywhere*
"I do so every friday to the USB3 disk drive that they did not supply using software that they did not supply."
I was tasked with recovering a very important USB drive where everything was deleted, and the off-site end user thoughtfully suggested an "unerase" utility, and how I could go about registering such software. Even though we worked for the same company, he clearly was not aware of the purchasing and capex policy of for software that did such frivolous things as data recovery.
Either way, I'm still not sure why he didn't store it on the company server, which is backed up nightly, of which he had a VPN connection into - I mean, he couldn't have been able to do too much work otherwise...
"Either way, I'm still not sure why he didn't store it on the company server, which is backed up nightly, of which he had a VPN connection into - I mean, he couldn't have been able to do too much work otherwise..."
Do. Not. Get. Me. Started.
There must be some universal unwritten rule employees learn AFTER their basic IT orientation that says employees should not use their assigned server folder.
Yes the servers go down sometimes and so does the network itself but is it that much of an onerous and Herculean task to make a copy? One for your PC and one on the server and then, you know, check the damn dates?
>There must be some universal unwritten rule employees learn AFTER their basic
>IT orientation that says employees should not use their assigned server folder.
1. $MEGACORP have never given me any IT orientation, basic or otherwise.
2. $NASTY_FACILITY_COMPANY have set my server folder quota to 5G 'because I am home based' and I have something like 500G of project data per year.
3. All project data folders on servers have been deleted in favour of a project management databse system that:
3A. Has no BLOB capability large enough to accept software projects (i could create a massive repository structure for hundreds of individual source files. If the PLC development environment saved them like that. It doesn't)
3B. I'm not authorised to access it anyway, even though I generate the actual working product. Itis full of sales memos, quotes, and copies of the brochures sent to customers. But no drawings, software, site reports, photos, signed acceptances.
From the other side of the coin, the company I work for recently (2012 I think) "upgraded" every employee's network folder from 1GB to 2 or 4 depending on your job role, but kept a requirement to hold 3 years of offline mail for legal reasons. Couldn't keep it on a server because our Exchange quota was still only 1G. I got lucky and was moved to a new unit after that was allowed to keep their rouge IT folks, so they kicked my quotas up to 40GB (personal share and my portion of a group share), and my maximum mailbox size to 15GB.
Before that, there was no point using the network folder because even compressed, I had ~18GB of data that wasn't squeezing into a 1GB folder.
I once lost some personal data despite having 3 different backup tools.
1) the Corporate desktop backup didn't backup my local Exchange folder because it in a different directory to the corporate standard due to an OS upgrade.
2) the TSM backup to the mainframe apparently stopped working because whomever was paying
the IBM license fee decided to stop and the admin cancelled all the backups without notifying me.
3) My Linux network backup failed because the building maintenance guys did a power/generator test and the Linux backup server did not boot
Moral: You can't have too many backups.
After all, fast, well-backed up samba shares have been around for years, and how is an old technology going to let us do our business in the modern world? Plus this is safe, it's in the cloud and backed up automatically, we'll never lose files!
Until we discovered, in a round about way, that whilst it's backed up, the best restore they can do is restore the ENTIRE workspace. Essentially, if you delete a file by accident (unless you explicitly turn on version control) you're fucked.
This is also, almost the case for Dynamics on cloud too...
"Oh you have some data missing, we can restore the entire database to a week last thursday, when you KNOW the data was there...."
"Uh Wut? You're shitting me, are there any alternative solutions"
"Well thats the easiest sir"
"Easiest for you, or for me thats trying to keep a f£!##@! business running"
"Well we could restore the database to a test environment & then manually export the data & reimport it into your current instance of Dynamics"
Man...the thought process at MS tech support is horrific sometimes...i still fundamentally like Dynamics though...
You can boost your beloved Sharepoint, but some want a physical medium for backup.
What kind of snooping-free promises do these services provide? None afaics.
No secrets outside work (a few, not secret to co-workers), but sure don't want to put anything in the cloud, where it is certain to be stolen if there is any interest.
We have a nice shared filesystem with a 5-tier permission system that is working relatively well here. automatically mirrored and backed up, nice. Only there's 5 TB of space available for ~700 of us. And No. Fucking. Quotas. So of course it's chronically full to the brim, and we have a locally-managed NAS box in a cupboard for our backups. Which had to be set up by the network guys so that it is accessible by the lusers without having to "configure" anything.
But it's locally managed, so the netops promptly forgot about it (2 years ago) and just switched the static subnet it was part of to dynamic... just before the holiday, as it were. When contacted, it becam (slowly) evident that they had forgotten everything about the config or the admin password that they set. We're headed to a factory reset as I type. As I am the cautious type I mirrored the part I have access to just before The Events, but a few of my colleagues did not see the need for it and are now well and truely screwed (that is, until I tell them I can pull the -single- drive out and restore from that, but I'm going to let them marinate a bit before I do that)
Pierre, if you're lucky the device *might* have a one touch backup button - throw a disk on of the right capacity, hit the button, leave it overnight.
Then plug it into a machine, scrape the data off. I think the Netgear and Syno devices both support this.
Question is, is the data worth the couple of hundred quid a 6tb disk will cost you?
Worth a quick google to see if the device does support it, and whether the backup files it spits out will be 'plain text' as it were - that is, just flat files.
Sadly, I suspect you may have already investigated that....also, tell your IT peeps to set up a local wiki server and start fucking documenting stuff!
(genuinely)HTH
Steven R
> Some businesses are really tight with the wrong budgets obviously.
Yup, I'd say that. Well to be honest there's perhaps 10 times that in bulk when you account for the RAID array and the offline backups, and it's all expensive 1st-tier drives, but it's still only 5TB accessible to the lusers. Endemic underfunding of research and all that...
5TB for 700 people? wtf?
Some businesses are really tight with the wrong budgets obviously.
No, it's probably sizing storage to meet needs. How many business letters fit in 5TB? How many records in a typical blob-free database? Remember that child benefit data loss a few years back - the entire database that's the core business of 3,000 people fitted on a couple of CD-ROMs.
That's par for the course these days - simple business records take next to no space by modern standards. It's media, video especially, that's driving storage growth now and the typical business has no need for a few thousand movies on their network.
> the device *might* have a one touch backup button
Yeah, we were kinda hoping to avoid that actually, especially as I have no idea whether the data in there has any value at all (Most of it doesn't, that I know for sure).
The good news is that I just went and plugged my laptop into the ethernet port of the NAS box, took a bit of fiddling to ifup with an IP in the right range but thanks to wireshark I can now talk to it in samba. The shared part is now saved, I just need to gather login/password info from the half-dozen of other users to check if there is anything of value on there before we can wipe everything and start fresh!
As most of them undoubtedly use the same password for banking I expect a bit of friction, but hey, if it's either that or lose their precious excel templates...
Or, "but if it's mirrored we don't need to worry about backing up to tape" (When it was pointed out that the weekly backup would take a month to complete)
Said about a multi TB storage array holding thousands of files that are quite important. Oh how we laughed!
Anon, but they know who they are!
Ahh, the classic I have a mirror therefore I have a backup idea.
Until it is pointed out firmly that all a mirror means is you will happily duplicate the missing file.
Or worse as I found many years ago, you duplicate the corrupted Master File Table, and lose everything on both disks.
My boss thinks that the answer to mirrors not being backups is another HDD in the same server that gets a copy of the SQL backups from the mirrored drives. I tried to point out to him that a fire would take out that as well as the mirrored drives so we need off site backups but he wouldn't listen.
Hopefully that office will burn to the ground some day taking our main SQL database and its *cough* backups *cough* with it.
I'm on holiday, and we had a DC crash yesterday, I was texted asking for driver cd's, DHCP config etc. I said the fastest way back is use the Back-UP! I have been asking for a test of the Back-up and restore procedure, but was assured that it will work as the back-up system sends an e-mail of completion.
I keep asking, a back-up plan is not a plan unless we can restore? Right? I'm waiting for an update, I'm sure they don't want to talk to me at this stressful time as I don't feel the burn like they do!
Yeah, kind of a squishy set up. This is what you get when the boss uses some sort of matrix to spread the work responsibility. I don't manage back ups, that responsibility falls to another colleague. But I manage a lot of systems he is responsible for backing up. I guess I have some layer of protection in case of issues just like this one.
This may be the case that changes our squishy set up.
If something is important it belongs on a server. If you lose something that was filed locally then boo hoo.
Notable exception: outlook PST files. These ***** things constantly corrupt when kept in network locations so HAVE to be stored locally. They are an utter joke/and constant irritation. Is anyone aware of an alternative for archiving mail in an outlook 2007/exchange 2010 environment? ...Other than storing emails in file sys.
I'd point out that nowhere in the story does it suggest the files in question weren't on a server. No one in a company big enough to have an IT department backs up users' desktops except as occasional exceptions to their backup policy for exceptional machines (we back up the one that has the ID badge printer and software for instance). Who has that kind of storage to throw away?
As for PST files, my alternative is a trash can. In my experience most people who have a need for PST files are saving emails that no one needs anymore. The prime example around here had three PST files because he'd run into the 4gb max file size in FAT32 when trying to store them on a flash drive (and blamed IT for it when he had to have multiple files, but that's another story). Looking at them it was pretty easy to tell the man had never deleted a file in his life. He still had emails from 1994 talking about whose turn it was to bring snacks when he retired last year.
I work at a very large company and we have Autonomy Connected Backup installed on everybody's laptops so apparently my 100,000 employee company considers the storage investment worthwhile. Granted CNB does have a very effective de-dupe component so the 100 or so TB needed is a drop in the bucket of our 4-5 petabytes of storage.
I feel your pain on PST files since big ones are more prone to corruption and people around here have years worth of email squirreled away.
Bad principle, in short. It's based on old methods and it's quite outdated by now, therefore making no sense: It's usually easier and cheaper to get something back from backup than trying to find the source for new copy. (Unless it's something than can be downloaded from a well-known source in few minutes.)
Our company is and was backing up everything, including every laptop, at least once a week.
But of course you need an intelligent backup tool for that, which doesn't store data as files, but 16KB (customizable, of course) binary blocks, where every block has a hash (for example MD5) and it's stored only once to backup media (3TB of storage for index and 800BG DLT tapes for actual data.)
Thus ~100 laptops with 60GB drives and almost identical content take only <500GB of space as 99,something% of the stuff is bit by bit identical in all of them and thus is
a) stored only once and
b) static content, not backed up every time we take "full backup": only hashes are updated up-to-date, no actual data is moved or stored.
Of course that means the backup media has to be reliable as we have only one copy of each data block. We were a bit more paranoid than that and we had 3 generations of sets of tapes, oldest off-site and rotated weekly. Supported by backup tool, of course.
Laptops had very small drives as general guidance is and was to keep everything important on server and only personal stuff on laptop. For that 60GB is enough.
Servers had about 40GB for everyone + space for projects as needed. Backed similarly as laptops.
Weekly "full" backup was about 3 DLTs in a robot handling 8 tapes because of identical block dropping.
I do agree that copying 100 copies of the same files doesn't make sense, but that method of backing up is, and has been, outdated for a long time.
My machine at home is decrepit (still love it).
At work, there is enough replication, so backing up is never a problem.
My decrepit machine at home has many bytes, but not enough to even save my media files lately.
I know I need a modern machine, planning to buy one by the end of this month (more trouble, finding out how to set partitions for Linux and earlier Windows on that)
... but the question is, what is the best form of backup storage for gigs for a home user? Blu-ray? Tape? Something else?
For a home user, a USB 3 external drive isn't a bad idea, along with your appropriate backup software of choice (on Linux, something like rsync isn't too bad, but you might want to rotate its destination dir to keep more than one backup on the external drive). I have an IcyBox USB 3 enclosure with a relatively cheap and fast Seagate 3TB HDD stuffed inside - seems to do the job OK and I get 100+ MB/sec write speed, which you really want if it's media you're backiing up.
You probably want something that will regularly nag you to backup (or can be hooked into your shutdown sequence) - remember that most of the time, the USB drive will be turned off, so you need something at least to prompt you to turn it on!
Even USB 2 is OK, if it's backing up in the background. And you do need to check they are actually working. My USB external drive wasn't, at first. It was OK sometimes, but lost connection randomly. A firmware update later and it's doing a fine job
A second HDD with or as a backup partition is also pretty good.
But both are vulnerable to anything really catastrophic.
Cloud backup is pretty much ideal for fallback home backups if you also have either or both of the local ones. (And aren't too worried about your stuff being "out there".)
I've been managing mine with Cobian backup (free). but Cobian has recently sold his source code - so who knows what will happen next.
I use three disks, a 1Tb, 2Tb, and a 3Tb disk. I plug them into my networked set top box and backup to that. Yes, media takes time, but I am not backing a movie every day.
I am a lucky Linux user, so I have an rsync script, setup in cron. Important files are in version control, I use git.
I set up a script to back the wfie's documents .... I use rsync to copy the lot over and then git to add new files, before I check the lot (new and modified files) in, again with git.
Honey, oh, you want your CV from yesterday, gimme a sec .... git .... there you are, honey.
Yes, I could teach her git, that way the repo would be on her system and backed up on the server AND she would be able to recover files from last checkin (like me), but she is not so much into computers ...
The data is then replicated over the three disks, again, with rsync. A year ago, a 1Tb disk failed, it was duly replaced by the 2Tb disk, now I have another 3Tb disk, as I will retire (from backups) the 1Tb disk.
I do not know, but Hans, Terry and Richard, thanks for food for thought. I tried to reply earlier, but must not have pressed the 'post' button.
Person who points out the idea of a physical print also has a good point (I thought the idea was the BASIC command at first). Screens surely do not do justice to good photographs ... but a print is not a backup, audio recording principle, always keep the original, even if the cleaned-up version sounds better.
Same applies to *good* digital video and photographs, I would think.
I'd vote for USB-drive as option number one:
Big enough to hold everything (assuming 1-2TB of data) , relatively reliable and fast, if you use it only for backups, like few hours in a week (those tend to overheat and cook the disk if used continously)
Blue ray might be next, depending on the amount of data: Blue ray burners are expensive(at least by my memory, haven't checked lately) and the disks are only 50 GBs (dual layer). That's not much in current computer environments.
DLT (tape) would be nice but those are quite expensive, both the tape and the HW. Quite fast (about 100MB/s) and very durable media but overall cost is quite high even the cost per MB isn't.
For small amounts of data, <50GB, DVD might be good option too as most machines have tools for DVD burning as stock, thus easy to use and no additional cost.
My boss, normally rational and intelligent, not least when it comes to backups, seems to have taken leave of his senses, having rejected the need to provide a separate backup system for Office 365. I'm not looking forward to the first incident, but I am working hard on my 'I told you so' dance.*
*AC because the 'I told you so' dance is strictly for internal mockery, not public embarrassment.
*AC because the 'I told you so' dance is strictly for internal mockery, not exposing the (L)user's shortcoming to their superiors, causing said fool to be denied a year end bonus, which will cause him extreme public embarrassment when the truth is finally revealed.
FTFY!!!
...here I'm synchronizing (one way) all files to a server on an hourly (or so) basis. Files that get deleted by accident is not deleted on the target server.
Then there's another server which is used for a monthly backup - since it's quick just to synchronize changed files (but not deleted) across. (Tape is taking too bloody long).
Then, finally, a tape backup to commit to long-term all the files/directories directly from the main server. A 5-day backup job.
Last time I had cryptolocker, I was able to recover files from the monthly server quick quick after zapping the cryptolocker'd PC. No need to wait for 5 days or so for files to be restored from backup.
And the backup server and tape streamer PC is in a different building (at last) than the main server.
I'm a lazy bastard, and have no wish to restore a file server from scratch.
"I have a series of 7 external disks which I backup onto. Monday, I backup onto disk 1, Tuesday, onto disk 2, etc. I keep each disk at a different location spread across the local area (friends, family, but none closer than 5 miles away just in case of a nuclear strike). In the morning I drive to drop off disk 1 (for example), and then pick up disk 2 on my route for that day's backup. And repeat with the next batch of disks the next day, and so on. As a fail-safe, I also do a complete backup to tape once a week, and post that tape to my brother in Australia, who stores it in his fallout shelter 20 feet under his house in the outback. This should keep my selfies and bank statements safe, and it's not that overbearing - the disks only cost £450 with a bulk discount, the weekly tapes and postage are just £35 a week, and I only spend £40 a week in petrol driving round. Plus, it's lovely scenery."
Or, you could use cloud backup where you can supply your own encryption key for a few ££ a month, and actually live your life.
"Or, you could use cloud backup where you can supply your own encryption key for a few ££ a month, and actually live your life."
And lose all of your backups once in a while. Or not being to able to back anything up as a) your network is down or b) the backup site is down or c) something between these is down.
If it were that easy, it wouldn't be work.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
