back to article Sysadmin hailed as hero for deleting data from the wrong disk drive

If the thought of another week at work has you down, worry not: The Register has another instalment of “Who, me?” for you to read, so you can enjoy another tale of errors made by someone other than yourself! This week meet “Wayne” who wrote to confess that “Once I accidentally formatted the wrong hard drive.” “I arrived on …

  1. Andrew Moore

    "Check, double check, then CHECK AGAIN!"

    Universally known as measure twice, cut once.

    1. Anonymous Coward
      Anonymous Coward

      A friend was renovating an old house. One of the new kitchen work surfaces was long and had a cut-out at one end for the protruding corner of a chimney breast.

      When it was fitted the cut-out was 100mm too long. My friend checked - and found that was what he had ordered. He re-measured and ordered a new one.

      It eventually arrived - was put into place. It was wrong by pretty much the same again - and so from then on the missing piece was a reminder of the need to check.

      Not long after he was cutting a piece of wood on the kitchen table - and relating how in his younger days he had accidentally sawn into a table itself. Sure enough he cut the kitchen table that time too.

      1. Chris G

        I haven't cut a kitchen table but while busy chatting to a mate I cut a plywood board with excellent precision and somewhat less precise was the cut right the way across my workbench.

        I spent the rest of the morning splicing it back into a usable bench.

        I started my working life as a painter working for my dad, we were painting some luxury offices a couple of doors along from Harrods.

        I had just mixed up a gallon of brilliant white emulsion to start on the ceilings put it on top of the scaffold and climbed up, as I did something slipped and instinctively grabbed out for support and of course knock a gallon of paint over on to the floor.

        Under the cotton dust sheet was brand new ankle deep, bloody expensive carpet and the paint went straight through, I spent hours scrubbing the carpet trying to get the paint out and then one of the clients came in, saw what I was doing and said 'Don't bother, we have just had an argument with the designer about this. He decided to repaint after he had the carpets laid, don't worry his insurance will pay for new carpet.

        Lesson learned , never overfill a buck and never perch it somewhere that is less than totally stable.

        1. Dave 126 Silver badge

          Cock up to triumph

          The Black and Decker Workmate was invented by a bloke who had just sawn through a chair he was resting a plank on. His error earned him a few bob.

          1. Peter2 Silver badge

            Re: Cock up to triumph

            I think most people do that at least once.

            Having sawed a (small) hole through my workbench my next project was a clamp on table protector with an integral cutting mat to minimise future damage.

          2. Stevie

            Re: Cock up to triumph

            I can't remember the official original impetus to build the Workmate, but it took years to arrive at the nifty machine you could buy 30 years ago. There used to be a history of the R&D with photos online.

            I have several Workmates. My original has seen much trench warfare.

        2. phuzz Silver badge

          I assumed that a proper workbench had a sacrificial layer on top that could be replaced* after a few years of drills and saws have had their way with it, but maybe that's just my dad being cackhanded and needing that room for errors.

          * well, flip it upside down and use the fresh side first obviously.

          1. Borg.King

            "I assumed that a proper workbench had a sacrificial layer"

            My whole workbench cost only $99 (homedepot.com, with free delivery in the U.S.). The whole workbench is sacrificial at that price.

            1. Anonymous Coward
              Anonymous Coward

              "I assumed that a proper workbench had a sacrificial layer"

              Damaging the bench in woodwork at my Secondary Technical School would have been a hanging offence***. There were small cutting boards which were used for the vertical impacts of chisels or drills.

              At a reunion some 45 years later there were still a few of the benches surviving. The contents of four 1960s state-of-the-art woodwork and metalwork workshops had been reduced to a small CDT? classroom that had usurped the junior toilets.

              ***dropping your pencil on the floor in Technical Drawing merited a few whacks from bunsen burner tubing. Two "offences":

              1) needing your pencil re-sharpened to the requisite chisel-point during class.

              2) you may also have fractured the internal part of the pencil's "lead".

              Before a TD lesson started there was an inspection queue. If your hands and nails were not suitably clean you had to go back down two flights of stairs to the cloakroom - with its cold water, bar of carbolic soap, and scrubbing brush. Understandable as you were going to be touching white paper for the next 45 minutes. Unfortunately our timetable always scheduled TD after metalwork - with its grubby filing and manual forging lessons. Less obvious was that our shoes also had to pass a "polished" inspection. Anyone failing that led to a ripple of boys in the queue nervously rubbing their shoe toe-caps against the backs of their long grey socks.

      2. Anonymous Coward
        Anonymous Coward

        My first experience with a circular saw

        saw me end up slicing right through the power cable. Thankfully tripped the house power but there was a loud bang.

      3. Paul

        I was helping a friend rip out his old kitchen and move light sockets etc.

        He drilled into the wall and the drill bit suddenly jumped, and he commented "oh, I didn't intend to go deep enough to hit the cavity". I knew that his kitchen wall was an internal single-skin, and I looked through the hole and said "no, you've drilled into the lounge".

        He thought I was joking. Then he looked through the hole and simply muttered "oh, bugger".

        Fortunately his wife wasn't leaning against the wall at the time, which she had been only a few minutes before!

        1. I Am Spartacus

          And that reminds me

          Of when my parents had the dividing wall removed and made in to a large through lounge. They employed builders, surveyors and decorators. Cost a bomb but looked fantastic.

          Next door, the young couple liked it so much they thought they would do the same, only being newly weds and therefore, by definition, poor they decided that it couldn't be that hard. So they started on the wall, and the old chimney breast also had to go, so he hit it with a kango hammer. Imagine my mums surprise when a brick popped out of the wall and landed in her lap.

          Sharp words ensued!

          1. Anonymous Coward
            Anonymous Coward

            Re: And that reminds me

            " Imagine my mums surprise when a brick popped out of the wall and landed in her lap."

            Our Victorian/Edwardian terraced house had a high shelf in the kitchen for storing big saucepans etc. It was heavy duty timber with its two wooden supports embedded in the party wall. As part of kitchen modernisation my father sawed the supports off flush with the wall. At which point there was a loud clang from next door as their shelf no longer had any restraint on its supports pulling through the wall.

          2. John Brown (no body) Silver badge

            Re: And that reminds me

            "So they started on the wall, and the old chimney breast also had to go, so he hit it with a kango hammer. Imagine my mums surprise when a brick popped out of the wall and landed in her lap."

            Lucky they stopped then. Chimney breasts are almost always integral to the structure and support of the house!

    2. Fizzle

      Or..

      In the case of e-mails:

      Read twice, send once!

      1. Anonymous Coward
        Anonymous Coward

        Re: Or..

        "Read twice, send once!"

        You should wait a few hours - otherwise you read what you thought you wrote.

        The problem is that you may be tempted to "improve" your email on the second reading. Only after sending it do you see the errors the changes introduced.

    3. YourNameHere

      Nope, I have measured twice and still cut wrong. :) If its important, I plan, measure 2-3 times, break for an hour or two so my mind has a chance to forget the wrong assumptions it's working on and then go back and measure from a different perspective twice and then cross fingers and cut. For back ups, back up to the NAS weekly, back up on line real time, and keep a spare HD that you back up a couple times a year. Some say this is to much, but when your wife is standing by waiting for her lost data or looking over your house remodeling job, you can NEVER be to careful... And the medication helps with the OCD.

      1. Stevie

        I plan, measure 2-3 times, break for an hour or two

        Uh-huh.

        Unfortunately my wife tumbled this some years ago and no longer accepts three-day estimates on shelf installations, citing "lead swinging" and using harsh words.

  2. Anonymous Coward
    Anonymous Coward

    Ghosted a PC the wrong way with my first use of the software. Ghosted a blank disc onto the System disk, fortunately there was another PC with the same software.

    1. Prst. V.Jeltz Silver badge

      It'd be nice if fdisk / ghost / the bios / all these types of programs identified the drives a bit more clearly , admittedly i cant think how , as they may not have volume labels at that point.

      I'vs come close to doing similar things as in the article and to this day , I unplug drives not involved in the operation for the duration to avoid confusion.

      1. Fatman

        <quote>It'd be nice if fdisk / ghost / the bios / all these types of programs identified the drives a bit more clearly , admittedly i cant think how...</quote>

        My PC does that, by identifying the drives by manufacturer and serial number.

      2. John Brown (no body) Silver badge

        "I'vs come close to doing similar things as in the article and to this day , I unplug drives not involved in the operation for the duration to avoid confusion."

        I came more than close. I did it to myself!

        I was experimenting with alternatives to Win98 on the second HDD. Rather than learning about and messing with multi-boot loaders etc. I just disabled the "C:" drive in BIOS where Win98 lived. Imagine my surprise when I got around to FreeBSD and discovered too late that it completely ignores the BIOS detection of HDDs and goes off and "discovers" them all by itself.

        1. Montreal Sean

          FreeBSD and BIOS disabled drives

          @John Brown

          That almost happened to me when I was trying out PC-BSD (I think).

          Fortunately I happened to notice more than one HDD got detected during then partitioning step of the install.

          I quickly shut down the PC, removed the install disk, enabled the disabled HDD and booted into Windows.

          Once I determined all was ok, I shut down the PC and disconnected the IDE cable for my Windows HDD before trying the BSD install again.

        2. cosmogoblin

          They do say that a person's ability with computers is in direct proportion to the scale of cockups they produce ...

    2. macjules

      Back in 1989 I had a request to transfer some very large graphics files to a 44Mb Syquest cartridge. Went to the client's office where I double-checked if the cartridge they handed me was ok to use since it would have to be reformatted. I was given the all-clear by someone who, as it turned out was only a freelancer working for the client.

      Luckily I asked him to sign his approval off as while he was searching for a pen the head of design walked in and demanded to know what I was doing with their main backup disk.

      1. Alan Brown Silver badge

        " demanded to know what I was doing with their main backup disk."

        Someone who'd obviously never heard of GFS backups.

        Yes, I know syquests were pricey back in the day but the point of having 3 sets of media is that if one gets toasted for any reason during the backup you have at least one working copy to restore from.

        I'm having the same argument with an "expert" at $orkplace who doesn't want to pay the costs associated with being on the central tape backup system. The last such "expert" ended up causing us to shell out more than twice the amount he was disputing, in order to recover his fried hard drive. It's funny how when they really cock up they expect us to both bail them out and carry the costs when they refused to follow our advice & procedures in the first place.

  3. GlenP Silver badge

    ACT Apricots didn't follow the IBM practice with disk naming so the HD was A: and the floppy B:

    Fortunately a format of A: would be noticed quickly (HD light flashing away, no activity from the FD) and could be rectified with judicious use of Norton Disk Doctor.

    1. J. R. Hartley

      Disk Doctor

      As an Amigaman, the words Disk Doctor still make me recoil in horror. Lazarus my arse.

      1. Antron Argaiv Silver badge
        Thumb Up

        Re: Disk Doctor

        Norton did OK by Disk Doctor:

        http://www.design-associates.com/projects/nantucket-marthas-vineyard/the-corbin-norton-house/

        1. MOV r0,r0

          Re: Disk Doctor

          Looking through those photos, there are so many places to fsck in that house!

      2. Black Betty

        Re: Disk Doctor

        NDD saved my backside after I made the mistake of typing RECOVER *.*

  4. tip pc Silver badge

    I hope he’s not still backing up disks before formatting.

    Granted it was harder to spot back then but in today’s world of right click format or using a disk utility app you can check it’s the right disk by logically unmounting it first then zap it.

    That said, I’ve accidentally deleted the running IOS of a remote switch, luckily I was able to replace it with a copy and it came back after a power cut a few weeks later, i’ve also deleted, replaced and reloaded a switch with the wrong updated IOS, luckily the switch was in the lab so tedious xmodem later it was all sorted, IOS for a 3560 on a 3650 or something easy to miss when doing 3 things at once.

    1. Alan Brown Silver badge

      "Luckily I was able to replace it with a copy and it came back after a power cut a few weeks later,"

      Doubly lucky, as some of the "supposedly" correctly setup cisco switches around here fail to boot up from power failures three times out of four - and when inspected, the tech support people can't find anything wrong with 'em (it's happening on multiple switches).

  5. Mystic Megabyte

    Ooops!

    TestDisk saved me once, I was so happy that I sent some money to the author.

    https://www.cgsecurity.org/wiki/TestDisk_Download

  6. Anonymous Coward
    Anonymous Coward

    The conversation went something like this:

    So you've got a backup?

    "yes"

    Format drive....

    OK where's your backup disks.

    Presented with a single 5¼ floppy.....

    Where's the rest of the backup?

    "This is it"

    Ahem in that case......

    Lesson learned to check for physical evidence of the backup first.

    1. Dave K

      I've seen that before. Someone who religiously backed up every day and presented me with a floppy when I asked for said backup.

      "How did you perform this backup" I asked?

      So they inserted a floppy, "format A:" and then once it was finished - backup complete!

      Result, lots of blank floppies, all nicely labeled - and all of course completely blank.

      1. Prst. V.Jeltz Silver badge

        "So they inserted a floppy, "format A:" and then once it was finished - backup complete!"

        You've gotta wonder what form of madness led them to arrive at that procedure!

        1. David Nash

          what form of madness?

          Probably following step 1 of a multi-step procedure.

          "Page 2? what page 2?"

          1. Prst. V.Jeltz Silver badge

            Re: what form of madness?

            ...or possibly some BOFH advice

        2. Anonymous Coward
          Anonymous Coward

          "You've gotta wonder what form of madness led them to arrive at that procedure!"

          I had a customer once who told me she'd earlier "debugged" her PC which had made it run loads better. I asked how exactly she'd done it - she reeled off loading up Word (probably 2003 at the time), something buried in the menu, which will have been to do with debugging VB macros.

          Anyway, she'd come across it, clicked on it and swore blind that made her PC run better. I explained it must've been a coincidence, but none of that was had. Weird how some users minds work.

          1. Anonymous Coward
            Anonymous Coward

            "I explained it must've been a coincidence, but none of that was had. Weird how some users minds work."

            Sometimes they are right - and the reason why an apparently coincidental change made a difference is only discovered by a deep insight into the root cause.

            My garden pond emptied itself overnight a couple of times recently. I had added more cleaned gravel and accidentally stamped on it. The obvious conclusion was the lining was punctured - but seemed to self-seal for a while.

            Then my waking brain said "what else changed?" The answer being that the "Boy on a Koi Carp" spouting water had been moved from the pool side - and was now sitting in the pond.

            Back-syphoning from a leak in the feed pipe under water when the pump was switched off? Logic said it was unlikely. Anyway - since the statue was raised to put the pipe clear of the water it hasn't happened again.

            Only time might tell - even submerging the pipe again would not necessarily replicate the previous syphoning environment.

            1. Anonymous Custard Silver badge
              Headmaster

              The placebo effect is just as applicable in tech as it is in medicine.

              And if it "resolves" an issue, don't knock it...

              1. usbac

                Placebo effect

                I would guess that most of us here have at one time or another, told a user that we made a change that will speed things up/fix a problem/etc?

                My experience is that it works more than half of the time. I will frequently try it first (based on who the users is, and past history). If the first "fix" doesn't do the job, I will go on from there...

          2. swm

            I have a 2-1/2 year old granddaughter and I wrote some programs for her to play with. She manages to find key/mouse combinations that are amazing that I never new existed. I've learned more shortcuts after a NO! or WHOA!

            Perhaps I could rent out my granddaughter as a tester to see if a system is truly hardened.

  7. Anonymous Coward
    Anonymous Coward

    My career taught me the value of the old adage "Do not burn your bridges".

    Colleagues and friends laughed at my paranoia - until the times it proved to have been very prudent for one of the "unknown unknowns" failures.

    1. JulieM Silver badge

      Or to put it another way .....

      The toes you tread on today might be inside the boots you will have to lick tomorrow.

  8. Roger Kynaston

    Not quite as serious

    but ...

    I worked for a local authority once. We had two V880s next to each other. One of them needed a patch and after installing it the reboot was taking ages for some reason, it had a lot of hardware to do a POST on, so some mug decided to reset it. You held the power button in for five seconds or so and it would go donk. All the disks were quiesced so what could go wrong?

    Trouble was I hit the button on the housing management rather than the payroll. Cue a whole department wondering what had happened.

    This was the days before strict change control so I got away with it by grovelling to the application lead who just laughed!

  9. Nick Kew
    Mushroom

    Partitions vs Slices

    I've been caught out by the different language of different OSs.

    I learned about partitions [mumble] years ago when I first installed a Linux to dual-boot with GamesOS (aka windows). FreeBSD also played well with that system.

    Many years later I took possession of a big chunky box with bare-bones Solaris installed. No docs. I needed to install a couple of other OSs on there, and I noted that it reported multiple unused partitions making that easy.

    So I went ahead. Memory is a bit fuzzy when it comes to sequencing, but at some point I came to setting up the bootloader and ... where is the original Solaris? Turned out those partitions I saw were in fact Solaris slices, and they don't play when confused with partitions. Whoops! No data lost - this was a new box - but a lot of faff.

    This was a time when there were radically different OpenSolaris versions: a relatively-stable one that only spoke slices, and a bleeding-edge one that talked slices and had an alien-OS-friendly bootloader in the manner of a decent Linux. And a great legacy of confusion, as the older version had nothing resembling a modern package manager, so software installs were utterly incompatible between the two, and inadvertently following a tutorial for the wrong one could brick a system! Endless trouble when $work required me to install stuff that only existed for the older version!

  10. Lee D Silver badge

    I've not lost data (yet!) because I rely on bog-standard paranoia.

    As in, I'd have removed the drive I thought it was, booted it back up, seen what happened / what was on the disk that was left in the machine.

    It's far too easy to break stuff, and relying on "undelete" tools is the refuge of the stupid. Even in work, whenever I Ghost or WDS something, I *always* remove the existing drive and put in a known-blank one before I re-image. Because then when the inevitable "Oh, I had some files somewhere that they shouldn't be and they weren't on the network and they're critical" happens, I still have their drive. And I keep it for 6-12 months before it ends up on the recycle-pile to be used as a blank one.

    When I bought a Samsung SSD the other year, I put it in my second drive-bay and ran the official tool. Which basically detected my identically-sized hard drive and showed me this:

    Source: Samsung EVO 850 1Tb

    Destination: Samsung EVO 850 1Tb

    I mean... if I could afford two, then I'd be over the moon! But in this case it seemed to be looking up the disk types by the size of them (for some reason) and - down to the byte - the drive I had was the same size as the SSD. And, no, there was no way to change it, or select another option. It put the SSD as the destination and then looked up the only other drive as the source, and decided that it was a Samsung.

    At this point, I could imagine a vast amount of things going wrong, so instead I cancelled out of the tool and made another backup (I had a backup, but in this case I sourced another 1Tb hard drive and bit-for-bit copied the existing drive to it. I'm not taking any chances with software activations, partitions, etc.). Once I'd done that, then I let the Samsung software loose on the drive.

    It worked, flawlessly, even though it still thought my Hitachi drive was a Samsung SSD. It copied everything over, removed all the swap files, shuffled partitions about and resized things, and when I booted up the laptop with just the SSD it worked first time as miraculous speed.

    But right next to me I had a full copy of the source hard drive ready to go. Because if it had somehow numbered those drives the other way round, I'd have been copying a fresh blank SSD over the top of an existing years-old configuration and data.

    I even still have that other hard drive around today. You never know.

    1. usbac

      No need for physical drives

      I do something very similar, but with a cool little tool called Drive Snapshot. I make an image of any PC I need to reload/refresh before blanking the hard drive. I put the images up on a NAS share that only I have access to.

      Drive Snapshot allows you to mount an image as a local drive. So, when the inevitable "I know I told you about everything I needed backed up, but I forgot this one extremely important file/folder/etc." situation comes around I can mount the image and retrieve the file. It happens all the time. Here it's usually the damned Excel macros that people don't think about.

      I could even recover the entire bootable drive if I really had to. I haven't had to go that far yet.

  11. Dodgy Geezer Silver badge

    This is the reason....

    ...that I went into computing.

    If I had become an architect and designed a bridge that had fallen down, I would be hauled up before a court and jailed.

    If I design a computer system that goes down, they pay me double to fix it.....

    1. Nick Kew

      You missed a treat there

      You should've become a Lawyer.

    2. Korev Silver badge
      Pirate

      Re: This is the reason....

      It depends what kind of a system you run. If you're supporting air traffic control or a hospital's PACS system then uptime is quite important.

      The nearest thing we have to an X-ray icon -->

  12. Anonymous South African Coward Silver badge

    Played around with OS/2 on a spare 2Gb HDD.

    Then I needed to use said spare HDD in production - made sure the spare is set as slave, and a windows (the one I was to copy from) is the master.

    Started up ghost, went my merry way, ghosted it over.

    And OS/2 booted up instead of Windows. Luckily a rebuilt-from-scratch was not a major operation, and was well-documented.

    1. Lord Elpuss Silver badge

      "And OS/2 booted up instead of Windows"

      There must be a BOFH article about doing this to a (l)user as punishment. If there isn't, there should be.

      Sidenote Am in your neck of the woods this week - Johannesburg.

      1. Anonymous South African Coward Silver badge

        Sidenote Am in your neck of the woods this week - Johannesburg.

        Format, reinstall or destructive format? :)

        Safe trip!

        1. Lord Elpuss Silver badge

          I would say:

          Format (charge the user for your time), then

          Reinstall (charge the user for your time), then

          Destructive Format (charge the user for your time, and for a new disk).

          That should stop them calling.

          And thanks ;)

      2. Prst. V.Jeltz Silver badge

        I do seem to remember a user / OS2 bofh article ...

      3. CrazyOldCatMan Silver badge

        "And OS/2 booted up instead of Windows"

        There must be a BOFH article about doing this to a (l)user as punishment.

        Some of us remember OS/2 with some fondness..

        If only it hadn't been owned by IBM - it might still be around as a viable alternative[1]. And not as an "how much?!" aftermarket OS from someone who has spent too much on the source code and rights and needs to recoup their costs..

        [1] Assuming it hadn't been borged by MS.

        1. usbac

          "And OS/2 booted up instead of Windows"

          So, it was an upgrade?

        2. RancidRodent

          Not owned by IBM?

          "If only it hadn't been owned by IBM" Well who would have written it then? The postman? The anti-trust agreement against IBM probably had a lot to do with them abandoning it - the reason the Microshite/IBM partnership broke down was Microshite wanted a presentation (over content) layer to compete with Apple - whereas IBM wanted a robust, dependable operating system which took a lot longer to develop. Obviously, Microshite won in the end because they inherited all of IBM's hard work which became the windows NT kernel and everything half decent from Microshite since.

  13. Waseem Alkurdi

    What about using that other recovery tool

    ... called the magnet, and you're the BOFH?

    Guaranteed to recover 100% of nothing.

    1. swm

      Re: What about using that other recovery tool

      At the company I worked for someone fastened a floppy disk to their door labelled "master disk" with a magnet. People would pass by and shudder.

  14. Doctor Syntax Silver badge

    Meanwhile whoever set up the slave as the C: drive has gone on to a long and successful career - or something.

    1. Rich 11

      A long and successful career in technical management, probably, where his idiosyncracies can be intercepted and corrected by his minions years before any harm materialises.

      1. Anonymous Coward
        Anonymous Coward

        his idiosyncrasies can be intercepted and corrected by his minions years before any harm materialises.

        Then. He discovers StackOverflow!

      2. Francis Boyle

        A variant of Murphy's Law

        If it can be done in a non-standard way* it will be done in a non-standard way. Corollary: never assume it hasn't been done in a non-standard way.

        *AKA a hack, in the old parlance.

        1. Anonymous Coward
          Anonymous Coward

          Re: A variant of Murphy's Law

          IIRC IDE disks could obtain their master/slave configuration in two ways.

          One was by explicit setting of the jumper on the disk as "master" or "slave".

          The other disk jumper setting was "cable". That meant a disk's master/slave setting was determined by which connector was used on the cable. Most PC bioses would boot quite happily from the supplier installed disk even if that was the "slave" - so adding a new disk to the cable made that new one the "master"

          I remember one model of PC that would not boot from a hard disk that was not in its mandatory position.

        2. Anonymous Coward
          Anonymous Coward

          Re: A variant of Murphy's Law

          "If it can be done in a non-standard way* it will be done in a non-standard way."

          Very true in the days when you had to designate each disks logical "geometry" of tracks and sectors in a bios setting - and there were several possible configurations possible for any model of disk.

          You had to remember to note the bios values before moving disks. If the bios battery died you were probably in trouble if you hadn't archived the geometry values on a piece of paper.

  15. Anonymous Coward
    Anonymous Coward

    MySql

    Always beware removing some of those large files buried in /var/lib/mysql to save space cos you can't be arsed to increase the disk size.....

    Just saying :-)

    1. storner
      Facepalm

      Oracle too ...

      Had a database server bickering about being short of disk space. Without knowing much of Oracle internals, I found some very large *.log files lying around and promptly deleted them - I mean, there's no need to keep those old system logs, right?

      So I learned the hard way what database transaction logs are. And how to convince Oracle to create a new set of transaction log files when starting up.

      Fortunately, it was a very quiet database.

  16. Cursorkeys

    I'm sure nearly everyone's done it...

    I've accidentally ddrescue'd with the target and source reversed and copied a nice, new, blank drive over the one with the data.

    Thankfully I had a tested backup and apart from the added faff, and a very red face, nothing was lost.

    So far one lesson was enough and I haven't repeated the error...

    1. Lord Elpuss Silver badge

      Re: I'm sure nearly everyone's done it...

      That's the way to do it.

      Having a good, tested backup turns this kind of situation from "You'll never work in the industry again" into a "No harm done, lesson learned for the future.".

  17. Gordon Pryra

    The importance of backups...

    Early on I was a field engineer for one of those software companies targeting doctors surgery's.

    NT4 was just being rolled out and it was my job to go to the surgery's and perform a final upgrade on their current workstations to ensure all sites were at the same level pre-rollout.

    Simple enough, except their OS was a propriety thing based around Ada, to assign more space to a hard drive you need to configure it, copy everything to disk and then format and restore.

    I was about 18-19 and got my job because I was a male who could type. Not much experience in "what happens if".

    I asked the Secretary if the backup worked last night and she confirmed it had, so I dutifully did my configuring and rebooting.

    Turns out the server had not been turned off in 5 years, so obviously the thing decided to die on me.

    No problems, I can drive to London grab a new one, now a few hours later, I learn all about the idea that you ALWAYS CHECK THE BACKUP WORKED YOURSELF.

    Seems the backup had never worked, the tape was ripped and all over the inside of the unit...

    5 years of the Doctors medical records down the drain with no backup.

    At least my actions gave a couple of temps a few months work typing in from the hard copy they had kept....

    1. Anonymous Coward
      Anonymous Coward

      Re: The importance of backups...

      You used "surgery's" twice as a plural noun. Do you have a fondness for the possessive ? The plural of 'surgery' is 'surgeries'. I hope your current skills required by your employer are a cut above your grammatical ones.

      1. Gordon Pryra

        Re: The importance of backups...

        You understood what I wrote yeah? not being English I am never sure I get my point across.

        I always assume the Grammar Nazis are able to put a good looking sentence together, but don't actually understand the words they are writing.,

        1. Lord Elpuss Silver badge

          Re: The importance of backups...

          We're all friends here :)

          Sarcastic, opinionated, pedantic, objectionable and sometimes downright abusive, but friends nonetheless.

          http://grammar.yourdictionary.com/punctuation/apostrophe-rules.html

        2. Anonymous Coward
          Headmaster

          Re: The importance of backups...

          > I always assume the Grammar Nazis are able to put a good looking sentence together, but don't actually understand the words they are writing.,

          Oh, I'm sure they understand the words they are writing down, just not the sentence that results. :-)

          Which is the perfect excuse to segue into a reminiscence: my first IT job was a student placement working for the Civil Service where one of the managers, a former chemistry teacher[*], so not especially versed in English, was able to write memoranda that were really quite beautiful in their prose and construction. Not overly long or flowery - just precise sentences that contained all that was required, smoothly flowing from one to the next. I was never able to achieve anything remotely so stylish back then, and certainly not now, where writing - proper writing - skills are no longer needed in IT.

          [*] I once asked him why he gave up teaching and he said that the final straw was when a pupil asked him if urea was soluble.

          1. Lord Elpuss Silver badge

            Re: The importance of backups...

            "[*] I once asked him why he gave up teaching and he said that the final straw was when a pupil asked him if urea was soluble."

            That's taking the piss.

            1. Gordon Pryra

              Re: The importance of backups...

              Why has this post not been given a billion +1's?

              I've just spit coffee all over my keyboard.

  18. frank ly

    Personal Tragedy

    About 20 years ago when I had my first computer from PC World, I decided to fit a new hard drive into the spare bay, to use as a data drive since the C: drive was getting full with all the music and video files that seemed to get downloaded.

    A simple job, easy to fit after some faffing around, configure it and make sure it could be seen by the OS. So I did 'format D:' and went to make a cup of tea. When I got back I quickly realised that something was wrong.

    You know how the 'C' and 'D' key are very close to each other on the keyboard? I lost about a year's worth of media file downloads and you'd be right if you guessed that I hadn't backed them up.

    After that I vowed that nothing like that would ever happen again. My OS system partition is backed up by regular partition imaging to a removable drive with a two image rolling backup on two separate disks and so is the personal data partition. My media collection is on a 1TB network drive and I maintain a separate copy on another drive that is only connected when making a backup. I do practice restoring the OS partition now and then just to make sure.

    1. Lord Elpuss Silver badge

      Re: Personal Tragedy

      These are the life lessons we (hope to) impart to our children/successors. But it never works; they need to make their own mistakes (like we did) in order to appreciate why things like backups, partitions, checking and testing are important.

      I once decided to consolidate all of my family's photos (10+years worth) onto one 1TB disk. Iomega, 3.5", braided USB cable, in a big old armageddon-proof aluminium desktop enclosure. 300 odd quid, if I remember right.

      Copied everything across from laptops, digital cameras, SD cards, USB sticks and the like, then deleted all the originals (otherwise what's the point?) and went on holiday for 2 weeks.

      Came back from holiday and the two-week-and-couple-of-days-old disk had a catastrophic head crash.

      Losing the photos was bad. Finding out that the warranty only covered the cost of a replacement disk (and not the cost of data recovery) was worse. Having to explain to my wife what had happened (when it was my bright idea to start with) was one of the worst moments of my technical life.

      1. Anonymous Coward
        Anonymous Coward

        Re: Personal Tragedy

        "These are the life lessons we (hope to) impart to our children/successors. But it never works; they need to make their own mistakes [...]"

        The most one can hope for is that the sagacity of your advice will become apparent when they encounter the situation - and thus their recovery will be much quicker.

      2. Anonymous Coward
        Anonymous Coward

        Re: Personal Tragedy

        "[...] then deleted all the originals [...]"

        Never delete anything irrevocably.

        In the old days of typing pools people learned - never tear/screw up a document you were throwing in the waste bin. It had a 50/50 chance of being the final good copy you had wanted to keep.

        1. Mark 85

          Re: Personal Tragedy

          In the old days of typing pools people learned - never tear/screw up a document you were throwing in the waste bin. It had a 50/50 chance of being the final good copy you had wanted to keep.

          More that a few of us keep our waste bin "hidden" from the cleaners as that is our "temporary" file cabinet.

      3. /dev/null

        Re: Personal Tragedy

        If you're using a proper camera with removable card storage, there is a simple way to do photo backups. When the card gets full, stick a new one in and put the old one somewhere safe. SD cards are cheap enough nowadays to consider them as write-once media and they certainly don't take up much space!

        1. 's water music

          Re: Personal Tragedy

          If you're using a proper camera with removable card storage, there is a simple way to do photo backups. When the card gets full, stick a new one in and put the old one somewhere safe. SD cards are cheap enough nowadays to consider them as write-once media and they certainly don't take up much space!

          SD cards are also fragile enough that you should not rely on them as ever-read-from-again media

        2. l8gravely

          Re: Personal Tragedy

          You do realize that these cards will fail over time? Just like CD-RW discs? They only real way to keep stuff like this safe is seperate RAID storage where you move the data to new media every five to ten years.

          1. Anonymous Coward
            Anonymous Coward

            Re: Personal Tragedy

            They only real way to keep stuff like this safe is seperate RAID storage where you move the data to new media every five to ten years.

            And who's going to do that?

            This is why we're now living in a digital dark age.

    2. hplasm
      Happy

      Re: Personal Tragedy

      "After that I vowed that nothing like that would ever happen again. "

      I called the data drive F:

      1. ricardian

        Re: Personal Tragedy

        About 25 years ago it seems that Microsoft decided that drive F: was special and could be used for all sorts of fancy debugging procedures. The result was that any drive allocated the letter F ran extremely slowly. I can't remember how I discovered this information but it quickly became well-known in the large Government where I worked (and had just scrapped OS/2 in favour of MS Windows).

    3. BonezOz

      Re: Personal Tragedy

      I had a similar issue about 17 years ago when Win XP was released. I had just "acquired" a copy of the "Devil's own" version and decided to upgrade my computer from Win 98. Trouble was my computer at the time only had a 6.4GB HDD and I was keeping all my kids photos on it. Now the installation went smoothly until I got into Windows and saw that I only had about 20mb of free space left. Immediately decided to reformat and reinstall Win98 until I could get a bigger drive, completely forgetting that there was about 3 months of baby photos sitting there. I was in the dog house for a while after that.

      But, I learned a valuable lesson that day, backups are important. Now all the photos and personal documents are stored in several places, in a backup file on a separate external drive, in my Google and OneDrives, on another drive that I carry with me to and from work, as well as another copy sitting on another computer.

      And I'd like to thank the author of this article, as he caused my paranoia to kick in and I've just created another backup file to sit in the cloud.

  19. Anonymous Coward
    Anonymous Coward

    Double destruction.

    Back in the late 1970s I was at university. The undergrads used a PDP11/34A with three RK05 drives, one holding the OS, one the student projects, and one for writing backups to.

    The RK05 drive holding all of the final year student projects died with a head crash. The damage was clearly visible on the disk - the head had crashed right through the magnetic media onto the metal.

    Time to call the engineer.

    The engineer wanted to check if the head had been damaged (d'oh) and so removed the crashed RK05 disk and replaced it with the first backup. Of course the drive head had been damaged, it crashed on the backup disk and destroyed it as well (again crashing right through the spinning rust onto the metal).

    Good job there was still one remaining backup although it was a day out of date.

    The engineer repaired the drive, tested it with his own RK05 disk, did a full format, and it was all fine.

    So being the "nice guy" he was, he decided to give the university his own RK05 disk as a replacement for the one he'd destroyed. And as we now only had one remaining copy of the final year student projects, he'd copy the last backup to the formatted blank disk.

    Unfortunately he didn't press the write protect button on the front of the drive now containing the student projects.

    And alas he got the source and destination the wrong way round, overwriting the last remaining copy of the student projects with a blank disk.

    I retrieved the original disk from the bin and have kept it on my desk for the last 40 years or so as a reminder to always do backups and to always do them properly.

    1. Lord Elpuss Silver badge

      Re: Double destruction.

      "I retrieved the original disk from the bin and have kept it on my desk for the last 40 years or so as a reminder to always do backups and to always do them properly."

      And to never let engineers near your stuff without having seen them work on somebody else's stuff first :D

  20. Anonymous Coward Silver badge

    RAID

    Replace failed disk in RAID array.

    Add disk to array. Start sync in the wrong direction.

    1. Steve Cooper

      Re: RAID

      I had an awful Apricot (what happened to them?) NT4 server once at work that the RAID controller went nuts and decided to break any drive that was connected to it. I thought it was a bit odd that all three drives failed at once, so replaced them and the new ones also failed almost immediately. Put the drives into another server and they didn't work there either - no idea what the controller was doing to them. Decided to scrap the controller and use NT4's built in disk mirroring - at least if the onboard SCSI controller dies then any old PC can read the disks in the future! Worked this way for years.

      Many many moons ago I did a del *.* not realising I was in c:\win directory on my dad's work IBM PS/2 P70 luggable he used to bring home at the weekends. Spent an hour or so with undelete making slightly educated guesses at the first character of each file! I got Windows 3.1 back up and running though in the end and my dad never noticed.

      1. John Brown (no body) Silver badge

        Re: RAID

        "I had an awful Apricot (what happened to them?) "

        I think they transformed into ACT then got bought by ICL/Fujitsu before fading into history.

  21. Vath
    Facepalm

    512 MiB of misery

    Back in my university days I couldn't quite afford to replace a slowly dying drive so every few weeks when Windows ME decided it was too corrupt to boot I'd reinstall it.

    Now, being a savvy if inexperienced CompSci student I kept a backup of my work on a USB stick, these frequent formats had taught me the value of such very quickly. I tended to leave my backup USB plugged in at all times when I was working at my desk so I could take very very regular backups of my project code while working on a PC with the inherent stability of a US President.

    Back then the first step in installing Windows was to format the disk, this is before it checks to tell you if your disk is going to be big enough or not. So away I go, performing the regular chore of reinstalling secure in the knowledge that I'd done this a dozen or so times already and the C drive is always the first on the list. It'll take half an hour or so to do that so off I wander to fetch a cuppa. On returning to my desk 10 mins later I find drive 1 nicely formatted and integrity checked, ready to install Windows, that was quick but oh well I won't complain. Hit next (was it space key or enter, I can't recall), select the freshly formatted drive 1, next again.... insufficient space!? Hmmm, reach for my trusty calculator and convert the huge number of bytes into something comprehensible. 512 MiB, but my C: is 40 GiB, it's my flash drive that's... OH F$@K!!!

    In hindsight I suppose it wasn't the platter that was failing but the controller, and when it finally failed completely the disk wasn't even enumerated by the BIOS. Lesson learned: always check that the backup media is disconnected when not in use.

    1. Old69

      Re: 512 MiB of misery

      "Lesson learned: always check that the backup media is disconnected when not in use."

      ...and have two back up copies. The most dangerous window of opportunity for SOD's Law is when you have the back up device connected to the PC.

      A friend called to say his PC wouldn't power up. I asked him to try again and tell me what was happening on the lights. What he then did was rock the PSU's mains switch several times in quick succession - from which I heard "A big blue flash!". Dead floppy, DVD, two hard disks etc. The only thing that survived was the cpu fan.

      1. Vath
        Pint

        Re: 512 MiB of misery

        Two 512 MiB flash disks on a student budget back in the early naughties would have been an indefensible indulgence!

        I've also fell foul of the power supply killer, as always my PC was plugged into an extension bar that was lost down the back of the desk, needing to unplug it I reached in as far as I could and tried to pull the plug. The plug got half way out before slipping out of my overreaching grasp, cue some buzzing noise as the contacts arc across the small gap shortly followed by a loud bang from the PC case.

        Lesson #2: unplug leads from the PC end first, then reach behind the freaking desk.

        Icon for what the disk replacement and redundant backup budget was actually spent on.

      2. J.G.Harston Silver badge

        Re: 512 MiB of misery

        I can't understand the number of times I've seen people who have somehow got it into their minds that the way to power off some equipment and power it back on again is to flash the power switch as fast as possible, presumably to try and get the equipment to not actually notice the power has actually gone off. When I was teaching summer school in the 1980s *EVERY* *WEEK* I had to clip some sod around the ear to stop them doing it. "Your parents have already paid 400 quid for you to be here, do you want them paying another 400 quid to replace that computer?"

        1. DavidRa

          Re: 512 MiB of misery

          I've had to do that when I've realised I've hit the power button on the wrong bit of kit (but before I let go). You used to be able to keep it on if you reapplied power within about half a second - sure you didn't want to do it all the time, but I saved my own bacon a dozen times.

          Then came ATX.

  22. js6898

    Well luckily going forward nothing like this whatever happened again at least on Windows 10 machines all you have to do is tell Cortana what you want to do

    1. Lord Elpuss Silver badge
      Devil

      You forgot the joke icon

  23. Anonymous Coward
    Anonymous Coward

    Back in DOS days, we had DoubleSpace running to compress data to a CVF. Even then users were filling disks. Got a call one day from a (slightly) tech-savvy user that they'd made more space by finding this HUGE hidden file called DBLSPACE.000 and deleting it.

    Oddly enough they were no longer able to find their data, and various error messages were showing on reboot.

    1. royprime

      Good old double space.

      I had a client that around that time found that his RLL controller would format MFM drives to a nice higher capacity. Didn't work properly of course and lost lots, but he was proudly showing me just how much more space he had on his drive.

      1. John Brown (no body) Silver badge

        "I had a client that around that time found that his RLL controller would format MFM drives to a nice higher capacity. Didn't work properly of course and lost lots, but he was proudly showing me just how much more space he had on his drive."

        Maybe it depended on the drive. I ran 20MB (boot drive) and 40MB (data drive) MFM disks as 32MB and 65MB for a few years with no issues.

    2. Lord Elpuss Silver badge

      "Oddly enough they were no longer able to find their data, and various error messages were showing on reboot."

      On the plus side, they did have lots of space...

      1. Anonymous Coward
        Anonymous Coward

        A KDF9 user complained that a well-established production application had started failing after a new version of the compiler libraries were installed on the mainframe.

        After some debugging it transpired that their Fortran program had a problem. A sparse two dimensional array was so large that some of the elements were overwriting code areas. With the old libraries these areas were luckily not required after that point in the program. In the new version they corrupted code that was still needed.

        The user was very miffed that his array was actually too big for the machine's memory.

      2. Anonymous Coward
        Anonymous Coward

        lots of space

        "clickety"

        1. Lord Elpuss Silver badge

          Re: lots of space

          “Let’s see, you have 4 meg free.” “Wow, 8 meg in total, thanks!” he says, pleased with his negotiating power. “No, 4 meg in total.” “Huh, but I’d used 4 meg already; how can...” I say nothing, it’ll come to him...

    3. BonezOz

      TL;DR I've done that to myself

  24. Anonymous South African Coward Silver badge

    Bah.

    Deleted a .VHDX by accident once - deleted the offending entry in Hyper-V, closed it all up nicely, and never made a mention of it - nobody mentioned anything missing....

  25. Anonymous Coward
    Anonymous Coward

    It's quite likely that my MSDOS foo has faded over the years, but I'm baffled by this story.

    If the secondary hard drive needed formatting, why on earth did this require physical removal of the drive? I don't ever remember this having necessary, and I've been using "PCs" since before they even had hard drives. Why wasn't a simple "FORMAT D:\" from the booted machine all he needed?

  26. jh27

    Shirley...

    the rule isn't "always ensure you have a backup of every disk before formatting" - the rule is "always ensure you have a backup of all data that is important"

    1. John G Imrie
      Unhappy

      Re: Shirley...

      You will find out that that one file you didn't include in the backup is the important one

  27. Mine's a Large One
    FAIL

    Sometimes just measuring isn't enough...

    Not exactly a fail like some of the others here, but still IT...

    I once asked a building contractor to cut a hole in a comms room floor tile, as there was an additional rack going in the next day. He marked it out and was about to drill a corner hole when I asked if it wouldn't be better to take the tile out. He opened the last cab in the line, put his arm through the hole under it, and proceeded to have a grope around. "Nope, there's nothing under that tile". After several further failed (increasingly heated) attempts at trying to convince him to take the tile out, he said I should stick to IT and he didn't have time to discuss it and drilled the hole.

    Nothing bad happened, but before he got his jigsaw blade through the hole, I tried again. He again had a feel under the tile through the other cabinet... "Nope, no need to take the tile out". He got about half way around the hole with the jigsaw before there was a loud bang, a bright flash, and the comms room (and an awful lot of the building) went very quiet... followed by the office manager knocking on the door... followed by the contractor's boss... followed by the electrical contractor... the supply cable he'd cut through was just out of his reach under the tile.

    Murphy's law in action.

    1. Lord Elpuss Silver badge

      Re: Sometimes just measuring isn't enough...

      “Murphy’s law in action.“

      That’s not Murphy’s law, that’s karma.

    2. Mark 85

      Re: Sometimes just measuring isn't enough...

      He was lucky he wasn't taken away in a rubber bag. You were lucky you didn't have to do the cleanup after he was removed.

  28. LeahroyNake

    Gas Axe

    I had just been sent to a new workshop and couldn't attend certain training course that everyone else had been booked on for the day.

    Instead of giving me the day off or just letting me laze about I was told to cut some old metal wheels / steelies in half so they could be hung on the wall to coil tubing/ wire etc

    The boss may have pointed to an angle grinder but I decided the gas axe was the better faster option and I would be finished much sooner. What I didn't realise is that a new floor with a very nice thick coating of oil resistant paint had just been laid.

    I spent most of the day buffing out all the pock marks left by the molten metal.

  29. Gene Cash Silver badge

    Grub rescue

    I did apt-get dist-upgrade on the 15th as I always do, then discovered the new nvidia drivers weren't compatible with the new kernel, so I downgraded to the last kernel version.

    Somehow I buggered my /boot, so I ended up in grub rescue. (not even grub - no tab completion, no help, no nothing)

    Normally I'd download a rescue to a usb stick, but I had no laptop or anything.

    I managed to get to the /boot on a backup drive and get to grub, and then to a booting system... about 20 minutes before I needed it to log into work. Yes, I was up most of the night getting it working.

    At lunch I realized I could have used a Raspberry Pi to make a rescue stick. Sigh.

    1. Trixr

      Re: Grub rescue

      I don't think it was *you* who somehow screwed it up. Almost exactly the same thing happened to me updating the Nvidia driver on the Solus distro. A distro that purports to seamlessly deal with graphics driver updates.

      For me, I didn't bother with recovery after I saw the grub rescue - I just wiped the Solus off my dual-boot laptop and reinstated the Windows bootloader. Yet another Linux distro that bit the dust for me. I get really sick of how fragile Grub is.

  30. John R. Macdonald

    Oh no!!

    A long time ago a mate of mine worked in a very high end audio/video store. An amateur group he knew asked him to duplicate the single demo cassette (I did say it was a long time ago) they had made, using the fancy equipment he had access to, before they negotiated their first contract with a record label.

    The battle scarred veterans reading this can easily imagine how well that ended...

  31. John H Woods

    Let me remove all these intermediate compiler files...

    rm *.o

    <sticky shift key>

    rm *>o

    OK, now I have just one file with one character in it. On the plus side, I have plenty of disk space.

    1. John Brown (no body) Silver badge

      Re: Let me remove all these intermediate compiler files...

      <sticky shift key>

      I nearly got caught by that on a users PC once. Never, never activate any of the accessibility options unless you have a very specific need to do so :-)

  32. Anonymous Coward
    Anonymous Coward

    I work for a shop that does a lot of computer repair, and based upon my experience dealing with customers that have lost (10+ years) mission critical or personally precious data because they couldn't be bothered to run even a floppy disk's worth of backup (metaphorically speaking), I swear that sometimes I feel like my backing up to 8 different devices (time variance between backing up my .pst file vs. doing a full system image larger, of course) still doesn't feel like enough.

  33. DanielR

    Amazing story and very funny hahaha.

    Must be Get Data Back. Not many can recover full paths and names as getdataback can.

  34. StuntMisanthrope

    Expand and Mount. f***k!

    1. You can't have enough back-ups.

    2. Never delete anything.

    #yesyoucanchargewhatyoulike #repeatafterme

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like