back to article Post-outage King's College London orders staff to never make their own backups

Despite losing a lot of user data from shared folders in October's mega-outage, King's College London is asking staff not to save work independently of the university's IT facilities. A month ago, departments across the university suffered "irretrievable data loss" when a 3PAR's one-fault-tolerant RAID Array kicked the bucket …

  1. wolfetone Silver badge

    In fairness to them they do have a point regarding the security of the independent backups. It's all too easy for someone to pop something on a flash drive and have it slip out of the pocket on to the back seat of a taxi.

    But, that point is only valid if they have a suitable back up procedure in place to begin with. Which they don't. So what do they expect people who have already lost data to do?

    1. Anonymous Coward
      Anonymous Coward

      And they TEST the ****ing backups.

      Twice I've lost data due to failed backups.

      That and the fact you get moaned at if you decide to shove 200gb of data on the servers to be backed up.

      Anon, because we are supposedly an IT company.

      1. hplasm
        Unhappy

        Re:And they TEST the ****ing backups.

        Backups never fail.

        Restores, however...

        1. SImon Hobson Silver badge

          Re: Re:And they TEST the ****ing backups.

          Backups never fail

          Err, yes they do. I've had both disk and tape failures during backup. I think you are confusing "backup failure" (as in fails and reports the failure) with "backup apparently worked but is unretrievable" (as in, reports success, but data isn't readable).

          Now, where had I got to in that book on advanced pedantry ...

          1. SomeoneInDelaware
            Facepalm

            Re: Re:And they TEST the ****ing backups.

            "Backups never fail"

            Horse hockey! For about 15 years of my working life, I had staff who's sole function was monitoring the backups, correcting problems with the backups, and TESTING the recovery from backups.

            Early in my career here, I witnessed the complete loss of our stockholder master files! The restores didn't work -- we had three of them on hand! We had to recreate them (keypunches in those days) from microfilm printouts.

            Cue the tune, "nobody knows the trouble I've seen...". :-)

          2. hplasm
            Paris Hilton

            Re: Re:And they TEST the ****ing backups.

            Confused? I'm not actually confusing anything. I may be confusing someone* though.

            *Seems like a few people... perhaps I should invest in some 'air quotes'...

            Of course they never 'fail'... Schrodinger's backup tape anyone?

          3. Dwarf Silver badge

            Re: Re:And they TEST the ****ing backups.

            @Simon.

            I think you missed the point.

            @hplasm's post was sarcasm that people think that the backups work but only ever find out that they don't when they try to do the restores - which is the whole thing behind this data loss incident.

            Suggest replacing the advanced pedantry book with "sarcasm in the English language" ;-)

        2. LDS Silver badge

          "Restores, however..."

          Once the IT team of a company I worked for tried to forbid me to backup our dev department servers ourselves (we still had some DAT drives used when we worked in a separate site, I needed to access the new server room to change tapes, and they wanted to forbid me the access, nor they agreed to change our tapes). They swore I didn't need it because they took care of all backups, and a separate backup was not needed.

          I asked my boss to ask their boss to perform a test restore to show us their backup worked flawlessly. I got the permission to access the server room again two days later...

          And I guess they never performed a test restore (I did one every six months, I was doing those maintenance tasks in my spare dev time....).

        3. Phil Skuse

          Re: Re:And they TEST the ****ing backups.

          > Backups never fail

          Clearly you've never experienced an Exabyte DLT Library. You lucky *******.

        4. Stevie Silver badge

          Re: Backups never fail.

          In what alternate Universe do backups never fail?

          I just spent a week diagnosing a very twisty method by which one particular backup was failing here, and a colleague "won his spurs" as a new hire consultant by diagnosing intermittent failures of backups positively as a function of the arm-parking algorithm of one of our robots.

          Even in the old days backups could and did fail, sometimes quietly because resources were not what they should be in terms of amounts available and sometimes spectacularly when the outer side of a tape reel decided to go for a bit of a walk while the reel was mounted and in use.

          Backups never fail. This is the sort of wooly thinking that brings Universities to their knees for days on end.

          1. Anonymous Coward
            Anonymous Coward

            Re: Backups never fail.

            > Backups never fail. This is the sort of wooly thinking that brings Universities to their knees for days on end.

            Stephen, do you realise that whoever wrote that was making what is called a joke, or you just wanted to tell us about your heroic little battle? You could have done so anyway.

            He was just pointing out that in many instances, the only way to know if a backup succeeded is to read it back.

            1. Anonymous Coward
              Anonymous Coward

              Re: Backups never fail.

              Stephen, do you realise that whoever wrote that was making what is called a joke, or you just wanted to tell us about your heroic little battle? You could have done so anyway.

              Hey, go easy on him. I liked this bit anyway:

              Even in the old days backups could and did fail, sometimes quietly because resources were not what they should be in terms of amounts available and sometimes spectacularly when the outer side of a tape reel decided to go for a bit of a walk while the reel was mounted and in use.

              :)

              1. Anonymous Coward
                Anonymous Coward

                Re: Backups never fail.

                > Hey, go easy on him. I liked this bit anyway:

                Yes, I agree with that. I've seen a guy in tears immediately following a vacuum system failure. That wasn't a back up, that was original, real-time measurement data that he was recording (and I do mean "was").

                Hard drive crashes are not what they used to be either.

        5. Jay 2

          Re: Re:And they TEST the ****ing backups.

          Tru dat!

          My own personal spin/mantra on that is having a backup strategy is fine, but what's your restore strategy?

          Where possible I try and stay away from managing anything to do with backups, storage or monitoring. As you always have loads of explaining to do when something goes wrong (or not to people's expectations). Though I'm failing miserably on this at the moment as I look after the monitoring setup and also check that the pre-tape backup setup is all kosher!

    2. Anonymous Coward
      Anonymous Coward

      Exactly - why the lack of information regarding what they had and now have?

      Really a RAID array going TITSUP should, at most, mean 1 day's data lost, and if your whole organisation depends on said array, then replication + snapshots + backups should be more frequent than that.

      Posting as an AC since our IT department have similar we-know-better attitudes at times, and also make dumb decisions (maybe due to cost of course) but that should be discussed more widely. Latest jape is the new networking will have almost ALL campus machines able to talk to all others. Just waiting for the first crypto-locker style infection with a worm/zero day and not just a hundred or so department machines where the poor fool opened the attachment will be hosed, but practically the whole organisation.

      1. GrapeBunch

        I foresee a return to a role for offline computers running obsolete OSes, pared down to a tiny number of applications, little or no security, communicating if necessary by sneakernet. There will be a place for the networked, secure, up-to-date computer, as a place to test USB sticks for malware before those sticks join the sneakernet. The machine the institution or company dumpsters today, you might wish tomorrow to have rescued. Those isolated machines will require their own localized backup strategies. With multi-TB self-powered 2.5" USB drives cheaply available, the strategy need not impress the Nobel Committee.

        1. Midnight

          Throw in some phones with cords, octagon shaped paper, and Edward James Olmos in a dark blue jacket and you've got something good there.

    3. Voland's right hand Silver badge
      Holmes

      You are slightly missing the point

      I am going to dust off my Chemistry hat and put it on. It is a bit tattered after 20 years not in use, but still does the job.

      So, with my Chemistry hat on, can you explain me who will successfully make something out of my raw NMR, Spectroscopy, X-ray diffraction, etc data.

      Even if I put my even dustier molecular biology hat on, while the raw data there is less obscure format-wise (it is usually just spreadsheets), unless you know what it is comprehending it is like trying to figure out hieroglyphics without having the Rosetta stone to start off with.

      Also, if it is REALLY cutting edge stuff, there are probably 10-odd people around the world which can make something out of your data and every single one of them _WILL_ send it back to you. That is how academia works.

      If there is any case where security through obscurity works it is advanced scientific data. Everyone but a select few "opponents" (which will return it to you anyway) will say: WTF, this is gobblygook and wipe it.

      Also, I have some doubts about the feasibility of idea of saving raw data on a "shared drive" or using a shared "enterprise" backup system exclusively. The guy suggesting it has no clue of the amount of raw data which can be generated by some instruments (NMR comes to mind).

      1. Flywheel Silver badge

        Re: You are slightly missing the point

        "who will successfully make something out of my raw NMR, Spectroscopy, X-ray diffraction, etc data"

        Apparently they will, and can in China. Hackers will steal chest X-rays of healthy people to pass screening tests that they might otherwise fail.

        1. Voland's right hand Silver badge

          Re: You are slightly missing the point

          You sir, do not know what you are talking about.

          Hackers will steal chest X-rays of healthy people

          versus:

          X-ray diffraction

          The former is personalized medical data which may or may not be of interest.

          The latter is a humongous raw data dump taken off a device which you cannot make sense of without running FFT and/or convolutions on it and relating it to other experimental data such as which f*** crystal did you put in the f*** spectrometer at the time. Sometimes, a few pints in The Eagle are required (so you get your head spinning in such a helical manner that you get a Nobel Prize as a result).

          In any case - I can bet that you wrote that without ever seeing a diffraction picture in your life. I suggest before you write s*** like this next time you have a look at this:

          http://undsci.berkeley.edu/images/dna/dna_forms.jpg

          and tell me if you can discern the DNA structure out it by just being a "Chinese hacker looking at it". If you can a Nobel Prize awaits you.

          This is by th the way the difference between scientific and personal or business data - advanced scientific data is naturally secure through its obscurity as it CANNOT be comprehended by an obeze lump in mama's basement. It takes decades of of sweat and tears in a university to be able to understand it in its raw form. Sorry, shit like Good Will Hunting in real life is in fact science fiction. Actually not even science fiction. Just fiction.

          1. Anonymous Coward
            Anonymous Coward

            Re: You are slightly missing the point

            > such as which f*** crystal did you put in the f*** spectrometer at the time.

            Why does it matter if it was a French crystal or spectrometer? And do you harbour such strong feelings against them?

            1. Doctor Syntax Silver badge

              Re: You are slightly missing the point

              And do you harbour such strong feelings against [the French]?

              Of course. It's a point of principle. Never heard of Brexit?

        2. This post has been deleted by its author

        3. Doctor Syntax Silver badge

          Re: You are slightly missing the point

          "Hackers will steal chest X-rays of healthy people to pass screening tests that they might otherwise fail."

          Darwin award candidates.

      2. lafnlab
        Boffin

        Re: You are slightly missing the point

        While that might be true of your chemistry and molecular biology data, it's not necessarily true for other people where you work (supposing you work for a university, as I do*). What about the department administrator who deals with personnel files? What about faculty who keep patient information for clinical research? If information like that was backed-up onto an unencrypted external drive, it's a huge fine waiting to happen. The college's policy is probably due to a desire for a "one-size-fits-all" approach where esoteric chemistry files are treated the same way as payroll files.

        *I have no connections with King's College, but the university I work for has similar policies.

        1. GrapeBunch

          Re: You are slightly missing the point

          Then procedures should require that the sensitive data backup be encrypted, whether it is stored on a USB drive or on the institution's own backup array. So we return to security by obscurity and obfuscation.

        2. Doctor Syntax Silver badge

          Re: You are slightly missing the point

          "What about faculty who keep patient information for clinical research?"

          Come to that, to what extent was KCH affected by this?

        3. Doctor Syntax Silver badge

          Re: You are slightly missing the point

          'The college's policy is probably due to a desire for a "one-size-fits-all" approach where esoteric chemistry files are treated the same way as payroll files.'

          This is the point. One size does not fit all and trying to lay down the law on the basis that it does it just plain wrong.

          Another aspect of this is that if you're going to insist on running the backups for the payroll you whould be able to demonstrate to the payroll management that your claimed backups are actually viable. As I've written in other threads the main requirement of a sysadmin or DBA is paranoia. You'd think that after such an incident the KCL team should be thoroughly paranoid now, even if they weren't before. From what's reported here there's no evidence of that.

      3. allthecoolshortnamesweretaken
        Coat

        Re: You are slightly missing the point

        You're not going to put on your wizard robe and hat as well, are you?

      4. Anonymous Coward
        Anonymous Coward

        Re: You are slightly missing the point

        They are probably more concerned about administrative staff making backups of the last spreadsheet of (say) student data they are working on, and then using something like Google Drive/Dropbox to "back it up".

        With any personal data in that spreadsheet, you've instantly violated data protection regulations by shifting the data outside of the EU (let alone the possibility of future data loss by the individual should they suffer account compromise int he future).

        Of course, there are also many Universities (I work for one - hence posting AC) that do have commercially sensitive data amongst their data (clinical trials, drug research, patentable/spin off company material) that could also lead to potential commercial loss if leaked.

    4. Anonymous Coward
      Anonymous Coward

      That's an argument against taking backups home, not against making backups to start with. A USB HDD sitting on the desk isn't going to fall out of your pocket on the way home.

      OK someone could take the drive but if they are in the building with unsupervised access to someones desk then you've got bigger security issues.

      1. midcapwarrior

        "OK someone could take the drive but if they are in the building with unsupervised access to someones desk then you've got bigger security issues."

        Either you assume all thefts are from "intruders" or you assume supervised access means someone always standing guard at your desk.

        I don't expect a College to have top secret level security, unless your Colleges are run completely differently that this side of the pond.

    5. LDS Silver badge

      Depending on how you backup (whole disks? just data folders?) there's also the risk some data are backed up to personal devices *only*. Then there's the sensitivity issue. But o course, as pointed out, users need to trust your company backup.

      For the matter our developer always work on local files - never on some remote shared disk. Why? Because if for any reason the VCS is lost and the server backups don't work, at least developers working on projects have a copy on their machines.

      1. Alan Brown Silver badge

        "But o course, as pointed out, users need to trust your company backup."

        You gain that trust by demonstrating that the process works and is tested. Not by fiat order from the top.

        Such orders frequently make people who weren't previously thinking about backups start panicing and generating their own, which is yet more data to possibly lose on the train.

    6. Yes Me Silver badge
      Paris Hilton

      What to do, what to do?

      Let's see... ignore stupid corporate policy and make my own backups, or risk losing everything due to incompetent IT department... which shall I choose?

      (Paris, because it's such a puzzle to decide)

      1. Anonymous Coward
        Anonymous Coward

        Re: What to do, what to do?

        Please tell me how you make your backups offsite or off the network when USB ports are blocked and read/write access to the C drive is blocked too, and emails are limited to 25 megabytes.

        1. Doctor Syntax Silver badge

          Re: What to do, what to do?

          "read/write access to the C drive is blocked too"

          Screwdriver access.

        2. Ogi

          Re: What to do, what to do?

          "Please tell me how you make your backups offsite or off the network when USB ports are blocked and read/write access to the C drive is blocked too, and emails are limited to 25 megabytes."

          I would zip the files up into one big one, split it into 20MB chunks, and send them that way. On Unix/Linux at least, it would not be a particularly hard script to write, nor to reassemble on the other side.

          However, if your place of work has a policy against sending data out, you can't use this method unless you want disciplinary action taken against you.

    7. Anonymous Coward
      Anonymous Coward

      Not the only university sector IT dept phuc-up, i-sol (aka ARSEsolutions) at one I know well lost many years of e-mails and data, thankfully as one of those affected by the £175,000,000 fire who had independent off-site back-ups, when most relied on extra drives attached to machines that got burnt with them, I refuse to keep all my eggs in one corporate basket. Even if ARSEsolutions new off site mega data-centre was properly secure and backed up some of their management team have little idea and even less interest in ensuring said backups actually are taken on a timely basis and checked for reliable operation post system failure.

  2. Nolveys

    This kind of reminds me of the movie "Deer Hunter".

    1. Anonymous Coward
      Anonymous Coward

      I was expecting to see "Die Hard"

      1. allthecoolshortnamesweretaken

        Obligatory xkcd is obligatory.

        1. Captain DaFt

          "Obligatory xkcd is obligatory."

          A bit like the joke about the old comedians home: I didn't even need to mouse over the link to know it was number 705.

          Am I too far gone to the nerd side to be helped?

          1. allthecoolshortnamesweretaken
            Happy

            Let us know if you think it's time for an intervention.

  3. Anonymous Coward
    Anonymous Coward

    That's kind of the way research universities operate...

    a loose amalgam of independent researchers coming from all over the place with all kinds of budgets and all kinds of old habits. There's no one-size fits all solution and woe betide anyone who tries to impose one. Have KCL's central IT department gone out to researchers and said "these are the services we offer, above and beyond the basics like network and email, these are the costs, these are the limitations, here's how you ask for what you need"? The needs of, say, a research fMRI group that have to crunch quadrillions of bytes of data, and the needs of, say, a psychiatric research group that have a small amount of company easily codeable but highly sensitive data, and the needs of, say, an atmospheric quality research group that gathers open data in real time from a network of sensors across London, and the needs of an archaeology team recovering or documenting valuable antiquities that require secure access from the middle of a Namibian desert, are going to vary wildly.

    1. Anonymous Coward
      Anonymous Coward

      Re: That's kind of the way research universities operate...

      My wife works at a research institution. It's exactly as the AC above explains. The in-house IT staff doesn't understand what each researcher needs, or care in fact. It's almost like they work for another company.

      I end up doing a lot of IT work for her research group. I'm not really thrilled about playing in someone else's pond, since as a network admin, I wouldn't be very happy if an employee's spouse came in messed with things here. But, they need the help and can't get what they need from their own IT department. I have had some discussions with their IT staff, and seem to have earned their trust. I think at this point, they're happy that they don't have to deal with things.

      1. H in The Hague Silver badge

        Re: That's kind of the way research universities operate...

        "It's almost like they work for another company."

        So like most departments in most large organisations then?

      2. Anonymous Coward
        Anonymous Coward

        Re: That's kind of the way research universities operate...

        > "There's no one-size fits all solution and woe betide anyone who tries to impose one."

        Indeed.

        > "Have KCL's central IT department gone out to researchers and said "these are the services we offer, above and beyond the basics .. these are the costs, these are the limitations, here's how you ask for what you need"

        Not speaking for KCL, but my experince of it is that simply doesn't work. Having tried to do this, you try to talk to the senior people in each faculty/school, who are generally too busy to talk to you, or too busy to pass on what you say to the researchers. If you put it in the researcher's induction packs, they won't read it (because no-one ever reads those). Trying to talk to individual researchers or groups doesn't work because you can't get enough of them together at the same time, and you don't have enough staff to do it. Rather than asking for things, researchers, who are intelligent enough to do it for themselves (and think IT will just say "no", or that because they can buy a HDD from PC World for half the price, IT are overcharging), just buy stuff out of their grant (sadly normally without thinking about long-term support costs). Thus, you end up with staff making their non-airconditioned office uninhabitable due to having a server rack they bought in it.

        So, "not good enough" but also understandable, from both sides.

        1. TRT Silver badge

          Re: That's kind of the way research universities operate...

          That's exactly my point. You can't please all of the people all of the time, so you tell them what you DO do, and then they can shop off your list, and you, unfortunately, have to support them and the shit they buy with their own money instead of trying to tell them how to do their research so it fits in with your CIO's unlovable strategy.

          For your example, you say "We have a choice of a number of small server rooms on the different campuses, and a large data centre in a swamp in Slough; you can rent rack space at £100 per U per year in Slough, or £150 in the middle of London (it's more expensive there you see). We provide a 100Gb datalink back to a vLAN of your choice ... etc etc. "

          But surely it's the CIO's JOB to know what researchers get up to and what strategies to employ to meet them, and what's realistic to provide and what's so niche they're going to be on their own? Isn't it?

          1. TRT Silver badge

            Re: That's kind of the way research universities operate...

            Ha! That did say unswervable strategy, but auto corrupt fixed it for me. I kind of like either version.

    2. Anonymous Coward
      Anonymous Coward

      Re: That's kind of the way research universities operate...

      Have KCL's central IT department gone out to researchers and said "these are the services we offer, above and beyond the basics like network and email, these are the costs, these are the limitations, here's how you ask for what you need"?

      Course they haven't But before sneering at KCL, how many organisations offer their users a proper choice? In the business I work for, some pea-brain in a large and Teutonic country has agreed some obscenely over-priced SLA with HPE, and there's no choice, no freedom. The evil axis of Procurement & IT have worked together to give us stale, expensive hardware, and overpriced, inflexible services that have been centrally specified without regard to specific business needs.

      Things that you could contract an SME to deliver for a few hundred quid in two weeks cost us hundreds of thousands, take forever (well, six months minimum) and often still aren't fit for the fairly simple purpose.

      All very well blaming Finance and PHBs, but in my experience IT departments can be a core contributor to the problem, particularly when the prohibit local or third party solutions, and thus give themselves a nice little monopoly.

      1. Anonymous Coward
        Anonymous Coward

        Re: That's kind of the way research universities operate...

        Any more TLAs you could have squeezed in there?

      2. small and stupid

        Re: That's kind of the way research universities operate...

        Thats not fair.

        Studying the Reg, you can see that many organisation have poor leaders, evil HR bods, antfucking accountants, dishonest sales staff, drunken marketeers, etc etc yet have somehow broken free of this shitfest of fuckwittery and recruited the cream of the IT world.

  4. Robert Carnegie Silver badge

    Let them use 7-Zip

    And an academically obscure password. Maybe written on Post-It note by a doctor. An unbreakable code.

    1. Chris King

      Re: Let them use 7-Zip

      Maybe written on Post-It note by a doctor. An unbreakable code.

      Just don't ask me to write it down, or it will be one-way encryption.

      At school, my careers teacher told me I should be a doctor... "King, your handwriting is so bad it deserves to poison somebody !"

      1. TRT Silver badge

        Re: Let them use 7-Zip

        I did the pre-medical illiteracy course too.

      2. Doctor Syntax Silver badge

        Re: Let them use 7-Zip

        "Just don't ask me to write it down, or it will be one-way encryption."

        Me too. Almost my first action after getting my grant cheque is-it-really-that-many years ago was to walk down the Strand to a typewriter shop & spend a whole tenner on a >2nd hand portable. I still have it somewhere but I think it's got woodworm - it has a wooden cover.

        1. TRT Silver badge

          Re: Let them use 7-Zip

          I still have a Smiths Corona daisywheel somewhere. Typed up many an essay on that beast, and a few dissertations too.

  5. AMBxx Silver badge
    Boffin

    Backup on 'New Technology'

    Windows NT 3.51?

    1. Anonymous Coward
      Anonymous Coward

      Re: Backup on 'New Technology'

      It's interesting to note, though, how it was good old tried and tested tape that pulled their 'nads from the bonfire when that there new-fangled RAID system fell-over.

      1. Number6

        Re: Backup on 'New Technology'

        In a place like that I would be using both. The purpose of the RAID array is to try to avoid any downtime due to hardware failures, the purpose of the tape is to provide an archive (chuck a tape set in the fire safe every so often) and a last-resort to recover as much of the data as possible should the RAID array go down irretrievably. I run a lot of my home stuff on a basic RAID1 system but I still rsync important stuff to a separate drive on another machine every night. I'd like to be able to afford a tape backup to go with it, but that's too much £££ to get a useful capacity now for home use so I rely on multiple hard drives on different machines. In the days of DAT tape capacities being bigger than hard drives I had a very good backup scheme that even managed a successful restore a couple of times after hard drive failures.

        1. Doctor Syntax Silver badge

          Re: Backup on 'New Technology'

          "every so often"

          i.e. at least daily.

          1. Number6

            Re: Backup on 'New Technology'

            I meant in addition to the regular backups. Most backup tapes get cycled through multiple erase/write operations, so something from a year ago might not exist in the current backup set, but an archive of what the system was like at regular intervals can be useful if something is deleted as 'not needed' but some months later it suddenly is required. Of course, with the penchant in the US to demand data dumps for litigation purposes, perhaps it's less desirable to maintain history if it might be incriminating.

  6. Anonymous Coward
    Anonymous Coward

    Auto encrypt devices

    Our IT setup automatically encrypts data written to external devices (HD also encrypted). So you can take backups locally (whether you should or not is another policy) but that data isn't going to be usable on non company devices. How secure or good this is I don't know.

    1. TRT Silver badge

      Re: Auto encrypt devices

      Depends on the circumstances of when you might need to restore it. If it relies on, say, certificates which have been destroyed, invalidated, lost etc. or passwords that have been forgotten or were only known to Dave in IT who has now left, then it's worse than useless.

      1. Mark 110

        Re: Auto encrypt devices

        The keys should be on a couple of physical devices locked in two different safes in two different buildings. Thats how I've always done it anyway.

    2. LDS Silver badge

      Re: Auto encrypt devices

      Unless the external hard disks uses its own encryption (I have one on my desktop now, it uses an hardware key to store the encryption keys needed at startup), or your whole system is designed to properly store (and backup!) escrow keys and certificates, you may end up with perfectly good but utterly unusable backups.

  7. Anonymous Coward
    Anonymous Coward

    Enterprise Endpoint Backup

    All these weird and wonderful problems and solutions above... what about Enterprise Endpoint Backup, designed for this very thing. Either cloud or on-premise.

    Oh I dunno, Code42/CrashPlan, Druva, etc?

    1. Doctor Syntax Silver badge

      Re: Enterprise Endpoint Backup

      The same questions have to be answered.

      Where does the data end up?

      How distant is it from the system it's backing up? (A copy of the data sitting next to the server protects against a disk crash, it doesn't protect against a fire.)

      How secure is it against unauthorised access?

      How quickly can it be brought back to site if needed?

      Above all, how often is restoration tested and with what results?

  8. Lee D Silver badge

    I specialise in working in schools that have experienced IT disasters, cleaning them up, restoring confidence in the system, proving it can run for a while (so I'm not just a fly-by-night merchant) and then moving to the next.

    I've done it for about 17 years, just not on the scale of KCL.

    There's good points all round here. Sure, you shouldn't be saving data which may come under the Data Protection Act on personal drives. That's a given.

    But you've destroyed user confidence here. That counts for an awful lot. What you SHOULD be doing its running around with a bulk purchase of, say, small NAS devices (which will be perpetually useful to you when you recall them) if that's what people are doing. You desperately need storage? Here, have a 12Tb array - that we can secure, encrypt, restrict, recall, replicate and then copy off when we're sure the problem has gone away.

    You've destroyed user confidence, and with it their obedience. Those are normally the points where someone like myself enters, as an unknown, and tries to enforce good policy while fixing the problem.

    My mantra is "I don't lose data". I will happily demonstrate the number of levels, checks, replicas, backups, etc. that I take to prove that to people. I don't lose data. You deleted stuff from last week? Here it is. Last month? Here it is. Last year? Here it is. You might not be able to see it instantly, but we don't lose data. You need to drum that home.

    But you HAVE lost data. And with the same IT people and the same equipment and the same suppliers you're trying to convince users that something has changed and will never happen again. That's an impossible task. Throw them a bone. You need to get back in their good books. I've already predicted that there should be a few pink slips up winging their way around the KCL internal mail, because this is just that serious. But you also need to throw them a bone, technically, to get them - and their confidence in your system - back.

    Literally, say, "We will provide you with multiple independent places to store your data while we make sure everything is back - they are under our control, we can still control the data on them for legal purposes, but here you go. There's a working area. You can safely put your years of research and teachings on there because you yourself can see that it's several different places, each independent and under our (yours and ours) control."

    It's expensive. It's huge. It's a big job. But if you want to restore confidence, it's a necessary step. Even "This network share is in The Strand, this one is in our other data center, they are independent, please feel free to copy to both". It's showing them that you care about their data (which is worth more than your job, I assure you), that you are letting them keep control of their data, but at the same time not encouraging hundreds of devices tucked under desks out of IT's - and therefore the Data Controllers - sight.

    I've been at my current place 2.5 years. Not a bit of data lost. Despite lightning strikes (literally blowing up a network switch), server failures, power tripping even UPS (crossed-phases), etc. Their data is still there. All the data that existed when I started, plus everything they've made since.

    My previous place, 5 years. Same. Took over a network that wasn't a network and then never lost a single byte of data. Was even asked to prove it at one point when a teacher claimed they'd "definitely" saved their old lesson plans - shadow copies twice-a-day going back months, backups going back years, replicas of those backups, and backup logs listing every file present.

    It's a core, basic, principle of IT. You are the curators of the data. It's up to you to preserve it, because nobody else will, it's up to you to prove that, and to ensure it applies to everything, and to survive a disaster - flood, fire, lightning-strike, even (for KCL) a potential bombing -.and to not lose things.

    But you lost it on a "routine" upgrade because you did not have backups in place sufficient to restore working order in good time. Literally a USB stick would have been better for most people. That's NOT running the IT properly, and hence why heads should roll.

    But to then expect users to throw all their research into ONLY your same systems again straight after that - after a huge, catastrophic failure of that self same system that wiped them out for weeks without any hope of restoration or working replicas- is dumb.

    Technically, ethically, personally, it's a dumb suggestion.

    Provide them with some confidence and make them trust you again.

    "Oh, you remember when we just accidentally lost all your children and couldn't find them for weeks? Well, we've changed nothing but you HAVE to give us your children again."

    We do NOT lose data. If you lost data - or sufficiently timely access to data that it makes no difference that it wasn't a total loss - you are not part of us. Not part of IT.

    IT do NOT lose data.

    1. Mark 110

      Lee - gave you an upvote for a great post.

      Not losing any data ever and being able to roll back to any in time in the last 15 years gets expensive, particular in large enterprises where the volumes of data are huge. Need proper policies and architectuure. Lets look at an example:

      A client I worked for had a requirement to retain all data relating to pension investments for the life of the policy (could be history going back 50 years plus by the time the the investments have matured and been paid out). They had the following data protection regime in place:

      - Dual data centres with synchronous replication (protects against DC Disaster).

      - in each data centre resilient Oracle database architecture storing to a high availability SAN

      - infrastructure was virtualised so inherited the resilience/recovery characteristics of the virtual environment

      - Weekly full plus daily incremental backups storing to HP StoreOnce disk based backup storage, replicated to the other data centre

      - Weekly full backups offloaded to offsite tape storage on a six week rotation

      - Monthly full backups offloaded to offsite tape storage on a 13 month rotation

      - Annual full backups offloaded to offsite tape storage on a 15 year rotation

      How many copies of the data were they storing? How much did that cost? OK so they had some good Dedup but it wasn't saving them more than about 50% capacity on that expensive HP Storeonce backup infraastructure.

      And what really did my head in is they had everything available on production. There was no proper archiving of old data. You couuld lose nearly all the tape without deprecating the data protection architecture very much. The applications manager who had been there 20 years remember a single instance of recalling a tape and the data the business thought they wanted off it was useless to them anyway. It was already on their live system.

      Working with the business and architecture we did some really good work to understand the business requirement properly, get a well thought through policy in place, and shaved probably a quarter of a million off thheir backup costs.

      1. Lee D Silver badge

        Agreed - you can go too far. But no restorable backup or independent architecture causing weeks of outage for a major university? You've not gone far enough, or you've gone too far in the wrong direction.

        You "shaved probably a quarter of a million off their backup costs." by reducing the number of backups, ultimately. So long as the number of backups is still sufficient to withstand any reasonably-expected disaster, that's fine, and sensible, and practical.

        But at the end of the day, backups and redundant servers, and redundant disks, and redundant storage, and redundant datacentres mean exactly what you would expect - paying for something that's probably never going to be called upon, to sit there and do nothing for most of its life, just in case you need it. And if the skill and time and effort and money invested in doing that is good, you'll quite possible literally NEVER need it.

        However, that's not a reason to cull backups, or skimp on backup infrastructure, or get sloppy, or do things stupidly. Because that's the circumstance that logically leads to PRECISELY needing such things, and then they won't be in place.

        It's like savings. The guy who carefully puts away a little here, a little there, constantly and doesn't touch it, probably doesn't need to worry even if he never gets to touch that money in his lifetime. But the guy who never puts away anything is only benefiting from his extra spending money until the first unexpected problem.

    2. Bodge99

      Just ***YES** to this.. I couldn't put it better myself!!

    3. Doctor Syntax Silver badge

      elReg, promote Lee to a gold badge for that post.

      1. Fred Flintstone Gold badge

        elReg, promote Lee to a gold badge for that post.

        Seconded.

        1. W.S.Gosset Silver badge

          Thirded.

  9. Paul

    this reminds me of how Toy Story2 was nearly deleted

    http://thenextweb.com/media/2012/05/21/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-and-again-for-its-own-good/

  10. Version 1.0 Silver badge

    Backups - yawn

    We go through this every year - they made backups so that's sorted ... until it all turns to shite. I have maintained for years that there are only two types of users out there - those who have lost vital data, and those who are going to.

    My backup strategy works like this:

    1. Daily backups in house - the server is backed up daily to a larger server that maintains 7 daily copies of the first NAS allowing me to go back seven days to retrieve data.

    2. Constant backups - the main server maintains a separate copy of itself in a different building.

    3. Slow backups - the main server keeps a separate image via a data rate limited link in the cloud - data rate limited so that I have time to catch a worm before it encrypts everything.

    Yes - I HAVE lost vital data in the past, it's not going to happen again.

  11. Jeffrey Nonken

    A critical part of doing proper backups is redundancy. One way to provide it is individual users doing their own. There are drawbacks, to be sure. But if IT are botching the job, whom else can you depend on?

    And vague reassurances are useless. I like Lee D.'s soapbox speech. Be explicit, be willing to demonstrate, be transparent. Be willing to listen.

    Redundancy, and layers.

  12. Alistair
    Coat

    Security fellow - secure backup!

    "Backup Path: /dev/mapper/vg01"

    *sigh* I never had the heart to tell him. Only because by the time I found out he was already headed out the door. Next group of security types were just as bad.

  13. Mark 110

    Policy, process, procedure

    Couple of great posts above from Version 1.0 and Lee D. But Volands Right Hand also makes excellennt points about the volume of data some instruments produce, whhethher its worth protecting and whether some of the data would be any use to anyone except the research team concerned anyway.

    So you need:

    a. a clear policy for data classification

    b. data classification types mapped onto clear backup, retention, security policies

    c. clear architecture for storage, backup, encryption, data loss prevention, intrusion prevention, etc to meet those policies (one client I worked with made you classify the data in every file you created - that metadata was then used by backend systems to determine the storage, backup, security applied to the file)

    d. lots of training to ensure academic staff understand what to do and why

    It doesn't sound like they had done any of that. Just bunged it on a raid config and hoped it would be fine.

  14. Anonymous Coward
    Anonymous Coward

    This is a reason I try to tell people how things are done. It's not good enough the company can recover from lost data and then put in a proper system afterwards. The reputational damage is done, data is everywhere, now on some very suspect devices and places (non company sanctioned cloud). They have spent time and money to recover data and put in another system and now more data is at risk of being lost and compromised than with the original system. Too many companies think they have employed robots who will be happy to be told if you took our advice and you weren't smart enough to now have a copy of your data after the last screw up then we don't have it either, so here's your P45. It's no wonder people take their own action, normally I would be hacked off but in these instances I don't blame them and see them as sensible people.

  15. A Long Fellow

    Staggering

    Somebody's following the ITIL playbook to the letter -- and completely ignoring that there are _humans_ involved in the equation. I studied this a few years ago* and concluded that the hallmark of a superb IT support operation is sensitivity to users' realities. It's not (just) about the tech, it's how well you support the people behind it. Support the humans, not the systems.

    Yes, I'm linking to it: http://hdl.handle.net/2381/9556

    1. Mark 110

      Re: Staggering

      Agree completely. And if you do actually follow the ITIL playbook to the letter it puts people at the front of everything. Just people get hooked up on thinking the tech is going to solve the people factor.

  16. Anonymous Coward
    Anonymous Coward

    This falls squarely on the CIOs desk

    There is absolutely no excuse for this in a modern organization and particularly in an educational institute. The CIO should be sent packing, if they haven't already been, and get someone in who knows about proper backups/restore and has a working Disaster Recovery Plan created.

  17. Number6

    Levels of backup

    I'm afraid if I worked there I'd be backing up my own data. It might just be sharing it between a couple of machines in the lab/office but I'd have more than one copy I could get my hands on without having to deal with the IT department. It also avoids embarrassment when you've accidentally deleted a file and would otherwise have to go ask them to restore it from the last backup (always a fun exercise, especially if it turns out they can't).

    1. Chris King

      Re: Levels of backup

      Backing stuff up between machines in a research group is one thing, but you still need to make sure you've got a backup somewhere else.

      What if you're all in one building that goes BOOMCRACKLECRACKLECRACKLE ?

      (I was there a couple of weeks before the blaze, and a colleague said "Oooh, I wouldn't want to be near those gas tanks if the place goes up !")

      I've also seen a research team lose access to everything when one of their number did something rather naughty, and Mr Plod seized ALL of their kit. They had all been backing up their work to each other's machines, and NOWHERE else. Fortunately, they got their machines back reasonably quickly, but not before they had to buy replacements and beg Plod to make a copy of their data to fresh disks.

      1. Number6

        Re: Levels of backup

        Yes, I'm thinking in the context of "proper backups are also being done". You'd be really unlucky if you managed to lose all your lab machines and the IT backup at the same time, although it could happen. I have been in the situation where the backup really is the only copy in existence and that's a bit nerve-wracking until it's fully restored.

        In the days when my data would fit on a DAT tape, I kept one in a locked drawer at the office that got swapped out every couple of weeks and also had one at my parents' house (a couple of hundred miles away) that I swapped out when I went to visit. Not 100% coverage but better than losing it all. I managed that once, along with probably many readers here, and am doing my best not to do it again.

  18. Uberseehandel

    Only in Britain

  19. adam 40 Silver badge

    Never known a restore to work

    I've worked in computing for 30 years and in the last 15 years I've never known a restore request to come up with any usable data.

    My main requests have been from data that I deleted by mistake, or something that has been moved on a server by an unintentional drag and drop.

    Occasionally I overwrite a file on a local PC that I am working on as local data.

    If the IT department had been operating a simple son=father=grandfather tape backup you would expect to get up to 6 months back, but the excuses for coming up with nothing range from "the tape is unreadable" to "the drive just shredded the tape", and "that drive isn't backed up".

    So I have to agree - if the data is vital to your job, or you spend more than a few man-days creating it, take your own copy too. It's the also useful when there's a network outage, power failure, etc, you can plug your drive into any laptop and carry on working. And you don't have to go to IT when you deleted that file by mistake!

  20. censored

    My company has axed all backups...

    I work for an educational establishment. They want us to use Sharepoint/OneDrive for Business, under the misapprehension that

    a) cloud storage = file backup

    and

    b) it's perfectly acceptable to create a ludicrous mess of a 'shared folder' which in the past two months now numbers 100s of files available online only, instead of just emailing the document you want

    To this end, we are no longer given backup drives for our laptops. Wonder what Microsoft will do if everything is deleted? My guess is bugger all.

    1. Number6

      Re: My company has axed all backups...

      The way to demonstrate cloud storage as a bad idea is a common-mode JCB fault on the internet feed to the site. All of a sudden, none of it is accessible unless people have taken copies. If they're shared files, you then get the chaos as multiple people work on it and there's merge conflicts when they all try to sync back to the cloud when the fault is fixed.

      How about the example where someone quit his job and, like a good ex-employee, wiped all of the company content from the Dropbox folder on his personal machine... Except he'd forgotten to disable the work account first so it deleted the cloud stuff too. Oops.

    2. Doctor Syntax Silver badge

      Re: My company has axed all backups...

      Show them the stories about this and Lee's post.

  21. Anonymous Coward
    Anonymous Coward

    "There will be lessons to be learnt from this incident," was the refrain of an apologetic CIO, Nick Leake, whose emails to staff were seen by The Register.

    Nick Leake

    https://uk.linkedin.com/in/nleake

    Price Waterhouse Management Consultants

    October 1987 – January 1993 (5 years 4 months)

    Graduate recruit to the Management Consultancy training programme

    Says it all really. University straight to Consultancy, never a day spent on the tools so to speak.

    It would be refreshing to see CIOs and CTOs who have some real world tech chops - but then no doubt the bottom line would suffer as things would be done properly.

    Honorary MBAs all round, why not? Can always blame the peons for getting it wrong and not learning the lessons (the only lesson being "don't make me look bad while I spout undeliverable promises then fail to deliver... wait no, I mean you failed to deliver, my plan was perfect, something went wrong in execution.....").

  22. tlhonmey

    Easy to fix...

    The solution is simple:

    "Yes, my department agrees to store its data exclusively on University shared drives as long as the IT department agrees to fund the replacement of any data they happen to misplace exclusively from their own budget."

    If they're confident enough in their backups to agree to that, then they're probably fine to use. And, if they're not, then you get paid to recreate the data. So you're good either way.

    There are two "rules of thumb" I give people for backing up important data. The first is that "data does not exist unless it exists in at least three places." The second is to calculate what losing the data would cost you, and then spend a third of that on building a good backup system.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021