back to article Data AWOL? Thank God for backup. You backed up, right?

Backup is a fundamental component of a healthy infrastructure. I admit backups are neither cutting edge nor sexy but they are important. It is an often-quoted statistic that of the companies that suffer serious data loss, one third go out of business within three years. Actually, it’s worse than that: 94 per cent of companies …

  1. Duncan Macdonald Silver badge

    Tell the management

    All too often the management do not see the importance of backups and very little resource is allocated. Tapes get reused until they fail (rather than being retired after a reasonable number of uses), the backups (when not automated) are performed by the most junior member of staff - as they cost less in overtime, DR is relegated to "we will think about it if a disaster happens", testing of backups is not done because it would cost time and money.

    Then the managers blame the IT staff when it all fails!!!

    1. Anonymous Coward
      Anonymous Coward

      Re: Tell the management

      > All too often the management do not see the importance of ...

      > Then the managers blame the IT staff when it all fails!!!

      That's why, in addition to getting an opt-out signed, I'd suggest keeping a sly copy of the paperwork where manglement don't know about it.

      Not that I'd suggest a manager would ever do anything so underhand as to try and change historical "facts" to their advantage. Noooo, I've never heard of a manager ever claiming they never sanctioned/refused whatever it is they are guilty of.

      Anon, because I work for manglement that would think nothing of "losing" such incriminating evidence. Mind you, that is sooo much easier when the process uses electronic forms so the chief culprit can just go into the database and change a 0 to a 1 (or vice-versa) to prove that they did ask for backups.

    2. Just An Engineer

      Re: Tell the management

      All to often it is the Backup Admin who fails to test the backups and it is usually due to not considering it important. As a Backup, Recovery and DR consultant, I have spent the last 7 years going to customers and examining the processes, architecture, and helping to design backup, RECOVERY as well DR Strategy.

      Management always overlooks the recovery part. They believe that its backed up so I am covered. However the BA needs to stress the recovery portion is important as well and usually fail to do so.

      When a BA has been in the same position for more then 5 years and I have seen them in place for as long as 20, then the resistance to change is incredible, this is called being to comfortable in the job. I have had BAs shocked when even before I have finished my interview with them, I can give chapter and verse on the processes they use or in many cases not. Some of these guys think they are the only ones doing it this way "this is what we have done for the last (pick your number) of years and it has always worked......" When people have been there that long the process is usually ignored if it even exists, and the documentation is non-existent.

      I have had front line management tell me to make sure I pointed this out in my final report so Sr. Mgt would see that I saw what he was telling them, and I did. It shows a definite lack of ambition being in the same desk in IT for more then 5 years and even that can be interminable. Especially when advancement and career paths are available in larger organisations. But they all believe they will be the boss wen the current one leaves ,and it does happen which only perpetuates the current poor practices.

    3. Burning Man

      Re: Tell the management

      Managers cop a lot of flack, sometimes rightly so, but they only go off the information they're given.

      Any IT department worth it's salt will have records of having sent spend requests off, with at least broad strokes implications if the money isn't made available.

      Businesses will be very adverse to company ending risk - it's up to IT to make sure the risks are properly understood, and if you fundamentally disagree with a mid-management mook making a bad call to save a few quid, go over their head. You'll probably find the CEO has a very different point of view on the matter (and with any luck will take some time to re-educate or replace said mook).

  2. CAPS LOCK

    Regular restores and comparisons, cycles be damned...

    ... as Ellen Ripley might say 'It's the only way to be sure'.

    1. TitterYeNot

      Re: Regular restores and comparisons, cycles be damned...

      Site Contingeny Planning List:-

      - Power Loss

      - Hardware Failure

      - Fire

      - Flood

      - Nuke the entire site from orbit - Erm...

  3. Little Mouse

    Backups - It's all in the name

    Every company I've ever worked for, from one-man-bands to full-blown Enterprises have understood at a basic level that "backups are necessary" and had a backup policy of sorts in place.

    Unfortunately, precious few of them ever put any real consideration into what the backups were actually for, so as long as backups were being "done", then that was that particular box ticked, regardless of whether they were suitable for purpose or not.

    The whole point of Backups is to be able to RESTORE data & systems when required.

    Maybe if they'd never been called Backups, but Restore Sets (or something similar) instead, more emphasis would have been placed onto the arguably more important half of the whole process.

    There's never a time machine around when you need one...

    1. Anonymous Coward
      Anonymous Coward

      Re: Backups - It's all in the name

      It's a regular mantra that I've fired at various purse-strung holders - "It's a backup system, not a restore system."

      However it seldom gets a lot of attention, not even the minimum it would need in order to be working restore system and certainly not what its importance merits.

      I tend to dedicate every unclaimed resource to some sort of BC / DR purpose - making guesses about what the business would miss most / need restored most quickly. I the absence of clear instructions (and proper funding) it's the best I can do.

      And like so many IT best efforts it's eventually self-defeating because people get used to the idea that "things never go wrong" or "you can always get it back".

      If I actually allowed one of the many disasters these people had brought themselves to the brink of I have no doubt that my hastily-appointed successor would get a much healthier budget to spend on this sort of thing.

  4. Andrew Moore
    Mushroom

    Ah memories...

    One of the worst calls I'd get when I used to do technical support for Psion's kit was usually from someone who had been storing all their "valuable data" in the internal RAM- which required power. They'd inevitably let the battery run down or fail to swap the battery within the 30 second time frame and puff went all their data. And I'd get a phone call demanding that I restore their "valuable data". I eventually developed a copying strategy which went something like:

    Me: "Don't worry, we can easily get that back. What I need you to do is get your last backup..."

    Customer: "But I've never done a backup."

    Me: "I thought you said your data was valuable."

  5. Kurt 5

    One large company I worked at had an IT department that went through the motions and didn't like to be questioned. In engineering we had a source server with a raid array that was managed and backed up by IT. The drives started going out and IT was a bit slow in replacing the drives. Time to rebuild from backups. Turns out they'd been doing incrementals every couple of weeks for 3+ years. Some of the tapes were missing and some were unreadable. Management's response was "IT did the best they could."

  6. Jay 2

    Having a backup plan is fine, but it's the restore plan you really have to worry about!

    It's all very well having the data backed up somewhere, but can you still restore it? Some backup products can be quite finicky when it comes to how and where they'll restore something. Meanwhile not having a suitable tape drive (or even backup software) handy may also be a bit of a problem.

    Personally I try and stay away from being too involved in backups, storage and monitoring, so not to be in the firing line when something goes wrong (and it will). Unfortunately that's failed miserably as I run the monitoring systems and also keep an eye on the backups to make sure they're actually running.

  7. Anonymous Coward
    Anonymous Coward

    One other thing.

    If you are involved in the restore process, make sure the media is write protected before being inserted or mounted.

    From experience:

    Restore request received

    Tape identified and retrieved from safe storage

    Tape inserted in drive

    Tape overwritten by a pending write looking for a tape

    F U C K ! ! !

    I'll never again forget to flip that little tab.

  8. 2Nick3
    Facepalm

    It's not just when a server is newly deployed.

    And don't forget about re-purposed systems, or those that have "functional scope creep." That file server that had an extra drive added, or the web server that had a database instance deployed on it. It really takes cooperation between the System Admins, DBAs and the Backup Admins to make sure that backups of the relevant data are happening the way they should be. And we all know how well those three groups communicate.

  9. Jamie Jones Silver badge
    Facepalm

    Unexpected backup failures

    I've posted the first part of this before, so please forgive the repetition.

    A number of years ago, I worked in IT support in London for a large company.

    An important sales division based in Manchester that dealt with the quick-turnaround stuff - pouncing on new opportunities etc. (I don't know the details, but apparently their work was more time sensative than usual) was affected badly by the IRA bombing. Turns out, local ops had a well defined backup procedure, which unfortunately didn't include offsite backups, because they thought their 'bomb proof' room and firesafe were sufficient.

    Replacement equipment arrived. The office was quickly assembled temporarily elsewhere, but all the data? processes, biddings, contracts etc.? No problem, we'll grab the backup tapes.

    Unfortunately, the building was a crime scene or whatever, and basically no-one was allowed near the place for weeks.

    Eventually, someone was allowed to retrieve the tapes. The backups were fine. The firesafe was fine. The bomb-proof ops-room was fine. But by the time anyone was actually allowed to retrieve them, the majority of the data was stale.

    ------

    On a personal note, I've been running my own hosted servers for about 18 years - basically just play things, hosting websites etc., and also hosting all my work and email and everything else, so I can always access everything from everywhere, even a borrowed mobile phone or tablet etc... Think 'the cloud' long before marketing types invented it.

    I did regular backups to 2 other servers, both in different countries, and with different providers.

    However, I got all 3 servers around the same time, which meant payment renewal was around the same time. That was all done automatically to the credit card, so no problem....

    However, unfortunately, about 3 years ago I had a very long hospital stay following illness. It was only 6 months later when I was getting back on my feet that I discovered that payment had been due, and had failed due to my credit card expiring/being renewed.

    Basically, all 3 services were shutoff by the time I realised, and despite desperate phone calls, all machines had since been reprovisioned.

    I lost everything from the previous five years.. emails, contracts, code, documentation, everything. The reason it wasn't longer was that I managed to recover a 5 year old full backup that was fully intact, off a machine at home that had died 2 years previously due to power supply failure. Ironically, I had originally been doing my backups to this home machine before I decided to switch to a more reliable backup solution...

  10. Anonymous South African Coward Silver badge

    Even backing up to a removable hard drive is not good enough - because cryptolocker.

    The best strategy is a three-pronged strategy. (Others may correct me as they see fit).

    First stage - backup to an external hard drive hourly.

    Advantages - quick recovery, especially undelete/undo folder move operation (can also use shadow copies)

    Disadvantages - Cryptolocker or other forms of data corruption can occur quickly.

    Second stage - weekly backups to a separate server in a separate building.

    Advantages : if main building burns down, you can recover quickly.

    Disadvantages : Your data may be anything from 3 to 7 days old.

    Third stage - backup to tape.

    Advantages : Can store for long periods

    Disadvantages : Not quick to restore, prone to failure if reused too much, need frequent testing

    Fourth stage : Backup to cloud (daily/weekly/monthly)

    Advantages : Totally offsite

    Disadvantages : Are your data really secure?

    So maybe we need to take a look at punch cards again?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2022