back to article The server's down. At 3AM. On Christmas. You're drunk. So you put a disk in the freezer

The On-Call inbox is full to overflowing, so we felt like letting another reader's tale of late-night weirdness into the wild. In this tale, reader “Nic” told us that “Back in the 1990s when being a web designer was a job and ISDN was considered hot stuff I worked as a sysadmin at an outfit strategically located above a guitar …

  1. Anonymous Coward
    Anonymous Coward

    Nic says he and his boss shared what sound like surprisingly uninhibited hugs at this point

    Hahaha - love it. Nice one.

    1. Pascal Monett Silver badge

      I have no trouble imagining the sense of relief that must have flooded the room at that point.

      The trip back home must have been like surfing a cloud.

  2. Sandtitz Silver badge
    Pint

    "DAT tape ... the data flooded back."

    That truly was a Miracle of Christmas.

    1. Known Hero

      Re: "DAT tape ... the data flooded back."

      Dat' truly was a Miracle of Christmas.

  3. Anonymous Coward
    Anonymous Coward

    That would be the worst Christmas day ever. I bet he spent the whole day in a state of anxiety that his phone was going to ring and the drive would pack in again.

    1. tfewster
      Facepalm

      Quite probable, once it's started acting up. The point of freezing/hitting it is to get it going long enough to back it up. If it seems to be behaving after that, maybe leave it running until a more civilised time to do a rebuild.

      I suspect the story has been shortened, but reinstalling the OS on a dodgy disk seems like a waste of time, especially if you don't know at that point that you have a data backup...

      Unless you're on triple-time for working Christmas day, of course ;-)

  4. Groaning Ninny

    Denmark Street?

    If it was the office of an ISP in Denmark Street, I remember a job interview there back in 2000 or so, and I think they were Sun based at that point. One of the selling points was the cool location and listening to people playing guitars. Turned the job down as a chunk of the renumeration was in shares... Still, they still seem to exist, so it mightn't have been a bad thing to accept, but the tide was certainly turning at that point!

  5. Nick Miles

    Nic behaved appropriately by ending his elbow. Repeatedly.

    How did he end it and why did he have to do it repeatedly?

    Was it a tricky bugger to take down?

    We must be told.

    1. This post has been deleted by its author

      1. Anonymous Coward
        Anonymous Coward

        Re: ending his elbow

        'Bending his elbow' ... boozing.

        1. gotes

          Re: ending his elbow

          I thought he was playing air guitar.

          1. MrWibble

            Re: ending his elbow

            I assumed he was pleasuring himself, until I read a little further on.

            1. Alan W. Rateliff, II

              Re: ending his elbow

              Indeed we discover he was providing much longer-lasting pleasure to himself, upon which the kibosh was put, like virtually all pleasures, by a ringing phone.

              I'll start:

              Dashing through the snow

              the drunk two wheels conveyed

              through the streets he roamed

              wailing all the way.

      2. Anonymous Coward
        Anonymous Coward

        Re: ending his elbow

        Can someone please translate that into English that I've heard of? (or "of which I've heard" for the pedants)

        It means he was on the lash.

      3. DCL

        Re: ending his elbow

        Should say "bending his elbow", as in repeatedly bending the elbow, as one would do when drinking.

        1. Anonymous Coward
          Anonymous Coward

          Re: ending his elbow

          One doesn't repeatedly bend the elbow. One maintains the elbow at a 90° angle and rotates the shoulder.

          Think Playmobile.

          1. Mark 65

            Re: ending his elbow

            You must have short forearms compared to the uppers - a normal person would be pouring it over their head doing that

  6. John Robson Silver badge

    There's even a Christmas miracle in this story, because Nic says that come Dec 27 his budget was increased by the cost of “a real system with proper redundancy and backup processes.”

    Why - they managed to recover from the situation without it - so we clearly don't need to spend all that money on what you call 'proper' systems...

  7. Anonymous Coward
    Anonymous Coward

    >Silly us: we blew a Christmas On-Call story months before the day.

    >Silly us: we blew a Christmas On-Call story months before the day.

    Come on, it's the end of September. The decorations will be up in the shops by the end of next week.

    1. xeroks

      Re: >Silly us: we blew a Christmas On-Call story months before the day.

      Our local garden centre had their decorations out last weekend. I think the manager there REALLY likes christmas.

    2. Rich 11 Silver badge

      Re: >Silly us: we blew a Christmas On-Call story months before the day.

      The decorations will be up in the shops by the end of next week.

      ...clashing with the Hallowe'en decorations and similar 'consumer opportunities'.

      Confuses the hell out of me. I can never remember which undead monster we're supposed to be worshipping.

      1. Zippy's Sausage Factory

        Re: >Silly us: we blew a Christmas On-Call story months before the day.

        I can never remember which undead monster we're supposed to be worshipping.

        Christmas is the one where we celebrate how the Easter bunny brought Frankenstein's monster back to life in order to put him on top of the bonfire on November 5th.

        I think.

        1. bpfh
          Trollface

          Re: >Silly us: we blew a Christmas On-Call story months before the day.

          And there was me thinking that it was when Jesus made the easter bunny lay chocolate eggs... or was that one for Easter?

    3. Steve Foster
      WTF?

      Re: "decorations will be up"

      Christmas Puddings are already out on the shelves.

      1. Fibbles

        Re: "decorations will be up"

        Christmas Puddings are already out on the shelves.

        I'd happily eat Christmas pudding year round so this is no bad thing. I'm not sure it outweighs having to put up with neighbours plastering their houses in fairy lights on November 6th though.

      2. DropBear
        Trollface

        Re: "decorations will be up"

        Wait - you mean this year Halloween and Christmas both fall on the same Black Friday...?!?

      3. Mutton Jeff

        Re: "decorations will be up"

        As are the cakes, advent calendars and obligatory boxes of Quality St / Heroes etc.

    4. goodjudge

      Re: >Silly us: we blew a Christmas On-Call story months before the day.

      My wife works for a Very Famous Store in central London. Their Xmas department opens in August to catch the tourist market...

    5. Chris King

      Re: >Silly us: we blew a Christmas On-Call story months before the day.

      My local Sainsburys had Christmas puddings and Stollen on the shelves in AUGUST, and one hotel was advertising for Christmas party bookings back in JUNE.

      At least they didn't play "Fairy Tale of New York" on my birthday (mid-Sept) as they have done in previous years.

    6. D@v3

      Re: >Silly us: we blew a Christmas On-Call story months before the day.

      Local supermarkets are already selling christmas puds. Also saw both christmas and halloween decorations in the same 'seasonal' area the other day.

      1. John Riddoch

        Re: >Silly us: we blew a Christmas On-Call story months before the day.

        My other half used to work as a chef. They had Christmas puddings in December. Any left over in January would magically become "Steamed Fruit Pudding" for the menu for the rest of the year :)

    7. Sgt_Oddball

      Re: >Silly us: we blew a Christmas On-Call story months before the day.

      Oh hush, the co-op round the corner from me has mini-eggs in already and a small selection of creme eggs (the tortured look on the staffs face when I enquired as to how this has beaten both the Christmas crap and Halloween guff said it all)

  8. Adam 1

    Shirley it would have been easier

    ... to tell them to call a guy to fix it like last week's printer guy and head back to bed?

  9. Doctor Syntax Silver badge

    “Sometimes it takes the CEO looking destruction in the face before arcane requests for budget have meaning.”

    Only sometimes?

    1. Swarthy

      Only sometimes?

      Yep. Othertimes, even narrowly missing destruction won't make 'em part with the money: "We were safe, without spending all of that"; and then when the destruction is not averted: "You let us down, you ruined us, why would we give you more money?!"

  10. Andrew Moore

    Wayne?

    "At least Nic seems to have been was spared Stairway to Heaven."

    "Denied" surely....

    1. Francis Boyle

      Re: Wayne?

      Well, he could compensate by buying this. Is it too soon to mention Rolf Harris?

  11. Keith 12

    It's cash in the till....

    Assuming it's the same situation as when I was involved some years ago. If an organisation is big enough, and has storage to cope, to buy at wholesale prices from a Chritmas product supplier - decorations etc, then the supplier will deliver in, say, April and there is no requirement to pay for the goods till December / January following. The manufacturer is happy as they can clear their warehouse and it puts less pressure on them to send out all the orders out in one go, as would happen if they insisted on end of month following delivery for payment. Naturally enough, the retailer(s), whose own warehouse is stuffed to the gunnels with "free" stock is happy to recoup as many sales as early as possible before payment is due and floods the shelves with stock well before the actual "season" starts.

  12. gaz 7
    Flame

    Freezing disk drives

    We had a long spell where end user PC hard disks would fail and this was before the time where everyone stored data on servers.

    We realised that the disks were failing after a few minutes. This obviously seemed to be a heat related issue, possibly bad silicon on the controllers.

    We surmised and were right that if we froze them to chill them down for a while, they'd work for longer & just long enough to copy the data off. We saved a few people back then, who quickly saw the merits of server storage :-)

  13. Anonymous Coward
    Anonymous Coward

    Same here

    In my case I discovered that memory chips eg SD cards often respond to freezing.

    I got the data off and backed it up *properly" this time, the problem is that every now and then someone claims that this works for HDDs. Sure, in like 1992 but these days it just makes things worse. You are better off sacrificing poultry at it than freezing/heating/whacking/etc the drive, it (usually) won't work but at least the professionals can get it back nearly all of the time.

    1. Alan W. Rateliff, II

      Re: Same here

      It is a yuge gamble, but a stuck spindle might be unstuck if struck at just the right moment with just the right impact, usually just before or immediately after you hear the spindle motor start trying to spin. I do not know if the heads move at all before the platters are spinning -- ISTR they need the spin to keep them "floating" above the media.

      In any case, I would much rather send the drive off to someone like DriveSavers but this is rarely (close to never) the first resort for customers. I found the pricing model to be the problem: DriveSavers and its ilk post recovery costs up front, whereas the time you will take lighting candles, sacrificing poultry (the Popeye's lunch suffices, IMHO,) formulating new incantations (mostly of curse words, old and new,) and such is an intangible until the invoice is received.

      I tested and confirmed this the last time I had to do data recovery in which there was no backup (not a regular client, thankfully.) I worked up an estimate based upon a worse-case scenario of the above-mentioned voodoo with no promise of results, then presented a specialized recovery quote which would not charge if no data was recovered. Off it went in next-day shipping.

      1. J. Cook Silver badge
        Go

        Re: Same here

        This. Very Much.

        I worked at one place that had several regional offices, all connected to the Home office which was in a separate city some 200 miles (and a 3 hour drive) away.

        The bulk of the users had mostly notebooks; Most were very good about keeping files backed up, either putting them on a CD-R or the remote file server when they were in the office.

        The boss of one of the regional offices kept his notebook *in* his desk, on a docking station, and came in one morning to find that it had crashed. I got called in, heard the Clunk Of Death, and asked if he had backed up his files to the file server sitting at the Home office. (this remote office had a full T-1 line there, which was pretty decent for 2004)

        He told me no, and by the end of the day I had extracted the drive out of the laptop, packed it in as much foam and bubble wrap as I could manage, and was driving it down to the local air freight depot to express ship the door thing off to OnTrack for them to work their $3,000 magic.

        And magic it was; there was a single file they could not recover, and it was an OS file. (I had already gone to the local computer store and bought a replacement drive and had rebuilt the machine whilst management was deciding if they wanted to spend the money to recover the drive.)

        To drive the point home of 'we know it sucks, but please back up your files to the remote file server', the recovered files were put on said server, and I pulled them across the line to re-populate the machine.

        (It was a small-ish company that didn't bother putting a branch file server at each office, although that would have made sense. *shrugs*)

        1. Anonymous Coward
          Anonymous Coward

          Re: Same here

          Many mumble years ago I came home from 'Uni one fine winter's morning to discover that my 500MB laptop drive was deader than Elvis and not spinning after less-than-careful handling courtesy of the luggage monkeys.

          Checked to see why and somehow the head had become hung up on the side of the platter.

          A few seconds work with a plastic tool precision made for the job and the head wasn't stuck any more.

          Got the important files off, put the old 120MB back in and copied them over.

          It worked for years after that!

      2. Down not across

        Re: Same here

        It is a yuge gamble, but a stuck spindle might be unstuck if struck at just the right moment with just the right impact, usually just before or immediately after you hear the spindle motor start trying to spin. I do not know if the heads move at all before the platters are spinning -- ISTR they need the spin to keep them "floating" above the media.

        I ran into that with Convergent MiniFrame which was fine when the box was running. Problem was if the machine needed to be moved as disks (Micropolis IIRC) had a habit of not spinning up after they'd cooled down. Not without help from good whack on the side anyway. Rubber mallet was useful tool back in those days.

        Lubricants have moved on since those days and these days if disks don't spin up it tends to be the motor packing up rather than sticky bearings. Likewise drives' power consumption is bit lower and they don't generate quite as much heat as the old ones.

  14. Tony S

    I had advised the site director about backups. He assigned someone to look after the job, but despite this, no tapes were swapped.

    As predicted the server failed; and the drives were the cause. No tapes swapped, despite my forceful admonitions, so no restore possible. I arranged for the tapes to go to a specialist recovery place; meanwhile, built a new server ready for the data. Got about 85 - 90% back at a cost of about £1400.

    You'd think that they had learned their lesson, but oh no. Still no tapes being swapped. So I ended up running a remote backup process, D-2-D-2-T. That seemed to be the only way to prevent a recurrence. Over the next 5 years, didn't lose a single file.

    I left the company and I'm told that the overseas parent company demanded they take control of everything. That included backups and they insisted each site manage their own process. About 2 years later, they had a server disk fail and no-one noticed. When a second drive failed, they lost all of the data, because no-one had swapped tapes!

  15. Version 1.0 Silver badge
    Thumb Up

    Further screw-driver acrobatics

    I had a customer with an 8" HDD (it was a long time ago) who couldn't get her DEC RSX11M system to boot and there was no backup. After listening to the disk noises, she got the idea that the disk just wasn't coming up to speed so she powered it down, took the cover off the disk and tried rebooting. Sure enough the disk motor wasn't getting the disk up to speed so she gave it a little help with her finger...

    The system booted up and she got all of the data backed up - kudos to her!

    1. Shades

      Re: Further screw-driver acrobatics

      I can vouch (sort of) for that kind of hackery... I had a slot loading DVD-ROM drive, back before most non-nerdy people knew DVD was a thing*, which eventually required a "bump start" to get it going; A quick push with a thin, but not sharp, object was enough to get it spinning and it survived like that for years.

      * I think even I only had 1 DVD movie for years though as they were still ridiculously expensive.

  16. CrazyOldCatMan Silver badge

    Banging on the desk..

    > disk remedy that involved freezing a disk and then whacking it hard.

    Sparc Storage arrays - I once had the (dis)pleasure of having to move a bunch of them to a new site. They had been up and spinning for years by that point..

    At the new site, about 40% of the drives refused to spin up - totally nuking the array. In somewhat of a panic (we had backups but they were in packing crates at the old office and the extra array that we'd got in case of hardware failure was doing the same thing) I phoned Sun (this was obviously in the days when they actually had people who knew what they were doing). The advice was along the lines of 'take the drives out, bang them on the desk, stick them back in'

    Which worked. I still lost a small proportion of drives but I had sufficient spares that I could bring up the array and let it rebuild itself.

    We got rid of those arrays as quickly as possible after that.

    1. Version 1.0 Silver badge

      Re: Banging on the desk..

      I've suggested that particular solution to many customers at one time - the particular gear had a steel washer fixed to an aluminium chassis and occasionally the oxide would go non-conductive rendering the unit non-functional.

      The customers would say, "But bang it? won't that break it?"

      I told them, "It's already broken - it can't get any worse"

      It fixed the issue almost 100% of the time.

  17. Burning Man
    Headmaster

    DAT tape?

    Brought to you by the Department of Redundancy Department

    1. Alan W. Rateliff, II

      Re: DAT tape?

      But dat tape, tho.

  18. Anonymous Coward
    Anonymous Coward

    In the early 90's, we had a Christmas power down due to electrical work. On returning in January none of our Sun workstations would boot up. Got them going in the end by taking out the hard drives and putting them on the radiators to heat up. Seems like the hard disk manufacturer had used bearing oil that was too thick and wouldn't allow the drives to spin up whilst cold.

  19. Thrudd

    The old ones need a lie down

    Many many moons and cpu generations ago...

    Had a batch of "recovered" drives that refused to consistently spin up and had intermittent data errors but were earmarked for a general use public file storage and message system. (BBS)

    Suggestion from one drive manufacturer tech was, since the "sleeve bearings" were obviously worn to set them on their sides, reformat and "never" move them again. Tinker toy to the rescue. The array lasted a few more years without issue.

  20. John Riddoch

    Fortuitously timed backups....

    BTDT - once had a script wipe out the mail spool folder (a combination of a script to remove mail folder for users no longer in NIS, /var/yp/securenets and a new subnet being put in which everything got multihomed on, including the NIS master/mail server). That was a bad time to realise we'd never reinstated the mail backups which were causing problems a few months back...

    Luckily, it was about a week after I'd taken a full backup of pretty much everything as an ad-hoc yearly backup thing and it was just after New Year (so minimal mails coming through). Managed to restore that and advise users how many mails they'd missed if they asked (by trawling the mail logs). That was a scary morning...

    After that, the script merely mailed us the rm commands to remove the mailboxes so we could sanity check it...

  21. crediblywitless

    Sometimes it takes the boss looking destruction in the face? It _always_ takes the boss looking destruction in the face, to consider spending money on backup.

  22. Anonymous Coward
    Anonymous Coward

    “Sometimes it takes the CEO looking destruction in the face before arcane requests for budget have meaning.”

    Not just budgets, but in our case (hence the AC) even making basic decisions. Even when we've had good advance notice of something (such as connectivity being terminated at a site) he'll procrastinate and procrastinate, and then when he realises that "yes, this really is going to happen and the fertiliser really will be flying off the air circulator soon" does he go into panic mode and expect everyone to work miracles to deal with it. Usually it's left to the last minute - eg if we've been told the lead time for a new connection is 12 weeks, he'll leave it till 11 weeks to "brown flying stuff day" before ordering and then expect the supplier to pull out the stops.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like