back to article It's always DNS, especially when you're on holiday with nothing but a phone on GPRS

Did your phone ring over the New Year? No? Then spare a thought for those unfortunates who remain answerable day or night to the dread trill of a panicked On Call. Cast your mind back to a time before the woes of today, when "Brexit" was a noise one might have made after a good night out and a few bad pints, Microsoft had yet …

  1. Admiral Grace Hopper
    Pint

    It was a quiet Friday night and I wasn't on call,

    so I'd been through the beer stage and was contemplating entering the vodka zone when the phone rang. It was someone from the Ops Bridge that I knew on a personal as well as a professional level.

    "I know you're not on call, but there's no answer to the bleep and we've got a problem with {SYSTEMNAME]. Can you come in?".

    "No way, I'm long past the point of driving legally. What's happening?"

    A period of drink-enhanced remote diagnosis and guidance followed with a successful resolution and resumption of processing after and hour or so. I poured myself a vodka and started to watch The Word (told you I'd been drinking).

    On Monday I asked my team leader, who was supposed to be carrying the bleep, what had happened. He clammed up. I asked his drinking partner if he knew what went on.

    "Oh yeah, we'd gone for a quick pint or two, the pager went so [TEAMLEADER] dropped it into his pint then went and bought another round".

    I learned a lot about professionalism that night. I still haven't really forgiven you, Brian.

    1. Phil O'Sophical Silver badge
      Coat

      Re: It was a quiet Friday night and I wasn't on call,

      Back in the 80s I was doing some work with another company office. They guy I worked with offered to take me out for a pint, but as he pulled his coat on he carefully searched through all the pockets. I asked if he'd lost car keys, but he said he was just checking for a pager. Seems that the favourite game for the unlucky on-callee was to 'accidentally' drop the pager into some random coat pocket in the cloakroom as they left for the evening. If you forgot to check your pockets before leaving you could find yourself elected...

      A good reason never to be the last to leave. See icon.

      1. Pascal Monett Silver badge

        That's no way to manage after-hours support. The manager should have a list of names with on-call dates beside it, and woes be to the guy who gets called and doesn't answer because he slid the pager to someone else.

        1. ArrZarr Silver badge
          Joke

          Welcome to On Call, the column in which the process is always followed, always professional and nothing ever goes wrong to highlight the flaws in the system.

          That being said, I cannot agree with your sentiment enough and if I were in charge, I would keep a tank of Piranhas for offender's danglies to be dangled in, OSHA be damned.

        2. Phil O'Sophical Silver badge

          That's no way to manage after-hours support.

          No argument there, but it was a small office and a small team, working on a prototype service. Seemed to work for them, and I was just getting paid to write the text-parsing & forms software. In Fortran ...

          1. Mike 16

            text-parsing & forms software. In Fortran

            I hope that was Fortran 4 or later. Lately I've been messing with some "legacy" (in the H.P. Lovecraft sense) FORTRAN2(-ish) code. I had forgotten just how close to completely uses A-format was back in the day (on a decimal computer, of course).

      2. Anonymous Coward
        Anonymous Coward

        Re: It was a quiet Friday night and I wasn't on call,

        As a long time holder of that name, I know it's always Brian's fault.

      3. Cederic Silver badge

        Re: It was a quiet Friday night and I wasn't on call,

        Pager, meet toilet.

        "No, I wasn't on call. Why would I have the pager? Oh. Well, go check my coat, it must still be in there, I haven't touched it."

    2. Oengus
      Pint

      Re: It was a quiet Friday night and I wasn't on call,

      On call roster - What is that. The Ops team had a list of names and numbers for the different application suites. I am sure they had my number on speed dial.

      I really don't miss being on-call.

      Beer for those of us who are no longer chained to the help desk.

      1. A.P. Veening Silver badge

        Re: It was a quiet Friday night and I wasn't on call,

        Beer for those of us who are no longer chained to the help desk.

        And stronger (if so desired) for those of us who are no longer chained to the hell-desk.

      2. Gene Cash Silver badge

        Re: It was a quiet Friday night and I wasn't on call,

        > The Ops team had a list of names and numbers for the different application suites

        That's the way it should work: the people responsible for the code are the ones that get bothered.

        It's amazing how quickly the code gets fixed when that procedure is followed.

        1. TSM

          Re: It was a quiet Friday night and I wasn't on call,

          Unfortunately there's no way I can forward my alerts (nearly all of which are caused by random Oracle webservice failures, usually returning a 502 Bad Gateway, the fix being to send the exact same request a minute or two later) to the people at Oracle who wrote the webservice code. So I have to deal with the fallout.

    3. Danny 2

      Re: It was a quiet Friday night and I wasn't on call,

      I was engaged to be married to the most lovely Amsterdammer but was put on call in the Netherlands by an incompetent US boss. Months before the project went live and was still in test. Other end of the country so we had to live apart. I had to cycle three miles to work whenever the phone rang. Cycling for a Scot, it's like bungee jumping if you are English, it's a wee bit of a shock to the system at 3am.

      Then I had to diagnose clustered NT Exchange servers, while I hadn't yet been given any Exchange course or even advice. The blue dots are doing this, the yellow dots are doing that, and are the red dots just in my eyes?

      Reader, I didn't marry her. Got the fuck out of dodge.

      I think I was only hired as a joke by the sardonic Dutch. He was fat as a gas giant, I was skinny as a rake, he was Denny, I was Danny. That's Dutch humour.

      He eventually put us both on an Exchange course in Edinburgh, asked me to pick him up at his hotel at half seven, drive him there. Which I did, but he was furious at me when I arrived.

      "I've been waiting an hour"

      "You told me to come at half seven"

      "But half seven is 6:30 in Dutch"

      "I know, but I'm not Dutch and this isn't the Netherlands"

      1. STOP_FORTH
        Unhappy

        Re: It was a quiet Friday night and I wasn't on call,

        You should never meet a German at "half past" anything either. Not sure if this applies to Austrians and Swiss?

        Icon - unhappy clockface

        1. Anonymous Coward
          Anonymous Coward

          Re: It was a quiet Friday night and I wasn't on call,

          It does apply in Switzerland

      2. Jou (Mxyzptlk) Silver badge

        Re: It was a quiet Friday night and I wasn't on call,

        Try a "Dreiviertel" oder "Viertel" Time in Germany. Dreiviertel = three-quarters, Viertel = Quarter.

        Dreiviertes-sechs, aka, three-quarter-six means, depending on the Region of Germany: 5:45 or 6:45, am or pm "as the mood decides".

        Viertel-Sechs, aka Quarter-six can mean, depending on the region of Germany, sometimes from town to town within the region...:

        5:15 (since it is already a quarter into six)

        5:45 (since it is only a quarter until six, the preferred interpretation of my region (ugh I hate it))

        6:15 (six and a quarter, which is my preference).

        There are things in Germany I don't like, so I always use "Sechs Uhr fünfzehn" which is the only safe was to express 6:15 so every part of Germany has the same reference.

        Luckily we are spared of that AM/PM Madness - why does the AM/PM notation not switch exactly at midnight and exactly at noon?

        1. jake Silver badge

          Re: It was a quiet Friday night and I wasn't on call,

          "why does the AM/PM notation not switch exactly at midnight and exactly at noon?"

          It's in the name, and from the Latin. AM is ante meridiem (before midday), and PM is post meridiem (after midday). Noon and midnight aren't considered AM or PM, they are just named markers of zero duration between the two.

          Try to remember, when this notation was first being developed the concept of minutes didn't exist yet, much less seconds.

    4. 9Rune5

      Re: It was a quiet Friday night and I wasn't on call,

      I learned a lot about professionalism that night. I still haven't really forgiven you, Brian.

      I could not help but notice your little down vote there pilgrim... Any chance Brian is a reader in these here parts as well?

  2. Anonymous Coward
    Anonymous Coward

    Hit once by this ok. Twice, more fool you

    After one New Years day spent trying to fix a system in London with the bit of wet string that was the internet from very rural Natal, because the person who was supposedly on call had gone walkabout and not taken the support phone with him, I made sure of two things from then on.

    1) If I am not on call over Christmas/New Year, everyone knows this well before and they all see that I'm leaving the 'On Call' Phone in the capable hands of those who are.

    2) I go to places where the only internet is via Satellite and as I don't have a Sat Phone... then tough.

    Worked very well until a new PHB came along and demanded that everyone be on call each and everyday over the holidays. When we asked 'What about you?'. His reply was 'I'm Sailing my boat from Lymington to Faro'. That didn't go down well so no one volunteered to be on Call.

    When the MD found out, our new PHB was gone, never to be seen or heard from again. Normal service was maintained and cover suppled.

    1. Khaptain Silver badge

      Re: Hit once by this ok. Twice, more fool you

      Exactly, if the Boss can't be reached then he's not a Boss.

    2. Colin Miller

      Re: Hit once by this ok. Twice, more fool you

      There's nothing quite like leading from the front

      1. Trollslayer
        Trollface

        Re: Hit once by this ok. Twice, more fool you

        Unless a cliff is involved.

        1. A.P. Veening Silver badge
          Trollface

          Re: Hit once by this ok. Twice, more fool you

          Even in that case, a good PHB is like a Polish mine detector

          1. Claptrap314 Silver badge

            Re: Hit once by this ok. Twice, more fool you

            If you had said, "Russian", it would have been accurate...

            1. Anonymous Coward
              Anonymous Coward

              Re: Hit once by this ok. Twice, more fool you

              Yet again Poland can hold their head up with pride. It seems they invented the mine detector, see Wikipedia.

    3. P. Lee
      WTF?

      Re: Hit once by this ok. Twice, more fool you

      Lesson: Change freeze kicks in a couple of days before people actually go away.

      People may not like it, but everyone is happier in the long run.

      Unless of course you have paid on-call and people are using it to top up their income. Most lack of on-call-volunteerism can be solved with cash.

  3. Will Godfrey Silver badge
    Angel

    No Service

    It's a good idea to find out where these areas are if you're on call - not that I've ever made use of such knowledge of course.

    1. sandman
      Devil

      Re: No Service

      A simple tactic is to tell everyone you're going to be out of contact, because you'll be spending most of your holiday exploring deep gorges/rain forests/coral reefs/etc with no reception. Many people will believe you because there is a strange belief that outside Britain and the US, mobile coverage is poor, when the situation is exactly the opposite. Then, when at your destination, turn the phone off and lock it securely in a nice steel hotel safe - just to be sure.

      1. Anonymous Coward
        Anonymous Coward

        Re: No Service

        I can drive about 25 miles from my house to an area of probably several square miles with no cell reception from any of the major carriers (not sure about minor ones) since it is an area where a lot of Amish live (I guess they don't rent their land out for cellular towers)

        If I wanted to say I was going to be out of contact, I could just say I'm going to the farmer's market there for the day I guess.

        1. Oengus

          Re: No Service

          I only have to go to my Rumpus room and I get no service. In my upstairs rooms I have marginal service from one of the three major carriers in Oz. In my lounge room I get minimal service from all three carriers. If I want good service I need to go outside to near the pool...

          Dead spots are everywhere here.

        2. Anonymous Coward
          Anonymous Coward

          Re: No Service

          Around here, all of the Amish carry cell phones because the bishop won't let them have landlines, and while they can't have mains power they can carry and use anything that is battery powered.

          Just don't ask where they charge them....

          1. Ivan Vorpatril

            Re: No Service

            Around here all the Amish have small solar panels on house roofs for that reason.

          2. Anonymous Coward
            Anonymous Coward

            Re: No Service

            Some of the ones around here (and I believe that dead spot in particular) are Old Order.

          3. Gene Cash Silver badge

            Re: No Service

            Ahhhh... the number of people I've caused to lose their shit when they proudly announce "I'm living off the grid!!!111oneoneone" and I reply "wot? like the Amish?"

            1. jake Silver badge

              Re: No Service

              What's really fun is pointing out the obvious when somebody claims to be off-grid, but has a 1,000 gallon propane tank and several hundred gallons of gas/petrol and/or diesel on hand at all times.

              1. The Oncoming Scorn Silver badge
                Pint

                Re: No Service

                A friend of mine in the Dalek building community, used to work out of a aircraft hanger, that backed onto Amish owned land.

                Testing his remote control\automated creations out at random intervals used to really piss them off, far more than the planes did.

                Icon - He runs a distillery now, so here's a toast to his new venture.

      2. phuzz Silver badge

        Re: No Service

        I know spots in the Cotswolds, not half an hour from Cheltenham (and GCHQ), with no signal. No need to leave the UK

        Or you could just turn your phone off. Many people these days don't seem to realise that that is an option.

        1. Venerable and Fragrant Wind of Change

          Re: No Service

          There's a long no-signal stretch on my semi-regular rail journey to visit my dad.

          No, not the coastal stretch under the cliffs and through tunnels - that's only maybe ten minutes. But on two mainlines and the stop where I change between them - only an hour and a half by intercity from London.

        2. katrinab Silver badge
          Paris Hilton

          Re: No Service

          Oxford city centre is also good if you want a no-coverage zone.

          1. Martin-R

            Re: No Service

            Glad it's not just me... EE always claims to have phone signal and solid 4G in Oxford but the internet crawls if it works at all. Soon as I'm on the bus out of town it all works again

          2. Jou (Mxyzptlk) Silver badge

            Re: No Service

            Oh, so Germany is not alone with its "99%" coverage, as the telcos call it. Recently the pressure on the telcos is rising since Germany made a new "report reception probems" App, which shows that it is not even 95% in cities...

        3. Mike Pellatt
          Happy

          Re: No Service

          Ah yes. That option. Was talking to the local Parish Council about my company's plans for FTTH into the village, and the possibility of our providing some public WiFi. One member commented that the lack of mobile coverage, WiFi and (even) decent wired Internet was what attracted some people to the place.

          Quick as a flash, I pointed out that there was always the "off" button. Sage nods of agreement all round the room.

        4. Montreal Sean

          Re: No Service

          When I'm not on call my work mobile ringer and notifications are silenced, from the moment my work day ends at 17:30 until 08:30 rolls around.

      3. big_D Silver badge

        Re: No Service

        I can just leave my phone at work - no reception there.

        1. Steve Kerr

          Re: No Service

          I can drive 20 seconds from my house, there's no mobile reception there - on any network, whatsoever. I'm not in the outback, being just several miles outside the M25. Quite bad really. Admitedly, it is an area in a small valley

          1. Tom 7

            Re: No Service

            Until very recently the only reception we could get for mobiles here was a 10 sq ft area outside (its always raining) a barn 50 yds from the house.Found this out when lightning took out the landline and internet and my dad got very ill. I didnt find the spot - some spotty 15yr old guest did!

            Always reminded me of the joke about the woman complaining she could see her neighbour naked and when asked by the police had to climb on the wardrobe.

          2. TRT Silver badge

            Re: No Service

            I think I know the place. Has an Ingress portal only hackable via the pub's WiFi.

          3. Anonymous Coward
            Anonymous Coward

            Re: No Service

            There is one particularly nasty dead spot in a town around here. I reckon the local factory is outputting some horrid interference. I'm tempted to check out what wavelengths...

      4. Jamie Jones Silver badge

        Re: No Service

        I would just say "my phone will be on the kitchen table, or on silent, switched off, or just won't be answered".

        No need to come up with excuses when I'm on my own time! (Not quite the total bastard, if a colleague did phone and I happened to hear it, and was in a position to help, I would, but don't expect me to be prepared for such a scenario!)

      5. Anonymous Coward
        Anonymous Coward

        Re: No Service

        My office at a minesite has no mobile reception, so I only get notifications of missed calls once I start driving to someplace else. The other fun aspect is trying to call out, basically wander up the hill out the back, and stand outside in the 40 degree heat and flies.

        Also very poor reception back at the accommodation village, again you have to have your conversations outside.

        All of which is a huge improvement over my early days, when you had to drive for 2 hours just to get to a pub with a payphone...

        1. Jamie Jones Silver badge
          Happy

          Re: No Service

          Of course, there was a payphone 10 minutes away, but it wasn't in a pub, right?

      6. Chris King
      7. TSM

        Re: No Service

        No need for any hijinks here - I do normally spend a week or so down at my dad's place over the holidays, and as they're in a semi-rural area there is indeed no mobile signal either there or on most of the trip down. (If you really desperately need signal, there are a couple of places on the property where you might be able to get a trace of it, if you wave the phone around enough and the wind is in the right direction. Had to call Microsoft for an activation code while I was there once, that was fun.)

        It's always interesting heading into one of the nearby towns from their place - and suddenly receiving a whole set of increasingly frantic text messages and missed call notifications, because something broke and my colleagues couldn't figure out how to fix it.

    2. jake Silver badge

      Re: No Service

      All kinds of places just North of San Francisco that have no service. If you need a quiet place to vacation, try Sonoma, Napa, Mendocino or Lake counties in California. Friendly people, and the best wine, beer and food in the US, as well.

      </shamelessplug>

      1. My other car WAS an IAV Stryker
        Facepalm

        Re: No Service

        Upper Peninsula (U.P.) of Michigan. Went snowmobiling with in-laws once, some April in the 200x decade, outside Munising. I assumed my Samsung flip phone could go a few days -- wasn't planning on making a lot of calls -- so I didn't bring my charger. Instead, it wasted power for hours trying to find signal while we were jetting around the woods, and died an hour into the ride home just as we got to the Mackinac Bridge.

        1. Charlie van Becelaere

          Re: No Service

          I managed to find one spot in the Taquhamenon Falls State Park campground where there was a mobile signal. If I moved a metre or so in any direction - nothing at all. Checked my work email (I know, foolish move) only to find a global "reply all" incident running rampant. Filled my poor Blackberry (yes, this was some time ago) with idiotic messages such as "please remove me from this thread" also replied to all.

          Still, some of the best camping, hiking, sight-seeing I've done in a long time.

    3. dak

      Re: No Service

      I'm regularly at my in-laws' house in Isle of Whithorn in deep SW Scotland, happily untroubled by communications from Three.

    4. Anonymous Coward
      Anonymous Coward

      Re: No Service

      My kitchen is a dead zone for at least major US provider (rhymes with 'Tint'), and I had 3 employers who issued company phones on that provider. 3 hours into a call that was going nowhere and I'd just start sliding down into the kitchen for a bit of a break...

    5. Tom 7

      Re: No Service

      I had a boss who complained he could never get me on my phone at weekends. I pointed out that I wasn't being paid for being on call and even if I was the fact he insisted Friday afternoons were serious drinking sessions - he once threatened to fire me if I didnt come down the pub where he was calling from NOW - it would have been a bit pointless.

  4. jake Silver badge

    In '95 ...

    ... my Satellite Pro (a 400CDT, if I remember correctly) triple booted. 4.4BSD, Minix 1.5 and this new-fangled thing called Slackware Linux. Slack had in the last year and a half taken over from BSD as my go-to OS. I wouldn't have been caught dead troubleshooting network issues with a Redmond based program loader pseudo-OS.

    1. Sandtitz Silver badge
      Mushroom

      Re: In '95 ...

      "I wouldn't have been caught dead troubleshooting network issues with a Redmond based program loader pseudo-OS."

      OK, I'll bite. How was Kildare's NT4 a pseudo-OS?

      1. Throatwarbler Mangrove Silver badge
        1. jake Silver badge
          Pint

          Re: In '95 ...

          Many moons ago, I bid on a contract at a un*x shop. I won the contract without a face-to-face interview. When I walked in on the first morning, the guy in charge of the data center looked startled & exclaimed "Where's your beard‽‽‽" ... Despite over forty years of un*x experience, I do not now and never have had a beard. Still makes me chuckle :-)

          Now git orf me lawn! (Beers all around.)

      2. jake Silver badge

        Re: In '95 ...

        Cutler's NT4 was an OS, true. Unfortunately, the Redmond marketing department managed to completely fuck up the "as shipped" implementation. This could be fixed in the field by competent IT staff, but rarely was, alas.

  5. Caver_Dave Silver badge
    Angel

    On Call / not On Call

    A long time ago, everyone in my IT group had to take a turn providing overnight and weekend cover for the haulage business (all their real work is done at night when they re-stock the supermarkets). I told the head of IT that there was no mobile coverage where I lived, but he insisted. So for four years, every month there were no responses to any calls for a week, when it was my turn.

    The company didn't collapse, I didn't get a bollocking, and I received the on-call supplementary payment. Everything must have just worked when I was on-call - I must have been a lucky talisman - I saw that the other guys logged lots of calls (unless that was a scam).

    1. big_D Silver badge

      Re: On Call / not On Call

      We had a guy who didn't have a telephone - at home. When he was on call, he'd toddle off to the telephone box 2 streets away and phone in at around 9 in the evening, before toddling back home and going to bed, if there hadn't been any problems. That was in the days before mobile telephones.

  6. Anonymous Coward
    Anonymous Coward

    My turn starting today.

    Thats me not drinking for a week :(

    And working everyday so cant use the "Deepest darkest Peru" excuse.

  7. Povl H. Pedersen

    Current workplace has servicedesk people on duty 24/7.

    People with knowledge are not on call. BUT, if things goes wrong, they will start calling managers, who then have to try to contact their employees. For now, there has always been someone who could help. And we are good at finding hacks to get things running until the right persons are available.

    I have been guiding people over the phone from holidays in Italy. But I never bring a PC on holidays, so people can not expect more than that. And if it is work, I will usually answer only when it is convenient. So I mostly get SMS messages telling me to read mail.

    BTW: After the on-call fees where suspended, all systems started to run much better during night. Now the people responsible did not want to get disturbed anymore.

    1. Anonymous Coward
      Anonymous Coward

      unofficial oncall

      $currentjob has a full shutdown 1st week in July. This year we found out that due to production backlog, we'd have limited production for that week (except for the 4th itself - USA here). Management was looking for volunteers to be available to provide support for any issues that might arise.

      Despite the late notice (1 day before break started) I didn't have concrete plans to be anywhere, so I told my boss I'd be willing to help. Late afternoon on our last scheduled day I stopped by his office to ask about how the'd contact me if they had issues (no existing policy for support). Boss said that they were instructed to call him and he'd contact the appropriate engineer.

      Then he gave me the closest thing to an evil grin that he can muster and warned me that he was spending the week at a campground with no electricity and minimal cell service, so I should expect he might not get any messages before the break ended.

  8. defiler

    Pager in a basement

    In the late 1990s I was doing web development for a property centre in Edinburgh, run for solicitors. If you've tried to buy a house in Edinburgh, you'll know them by their 4 initials. Typically for web dev work in the 1990s, a large part of my efforts involved editing images so that (for example) maps were as clear as possible, but would still fit in a 16-colour GIF and zip down a 28k8 modem in moments. Then there was tearing all the crap that FrontPage insisted on cramming into the HTML. Finally there was the tricksy bit of hooking it into the live database of properties for sale and drawing through the schedules, photos, searching on various criteria etc etc. It was pretty groundbreaking in its day, and to be honest I'm quite proud of it (except that it didn't actually verify if there was central heating in the property, as there was a bug in that search that I only realised much later, so apologies if you bought a cold home in/around Edinburgh!)

    It's all been ripped out and replaced by now, but all the while I was working there, I was in the basement of their George Street head office. There was a Seattle Coffee Company shop also in the basement, and I didn't need to surface into daylight for hours at a time. And all the while my pager was reassuringly silent. It went mad when I emerged blinking into the sunlight, mind, but that wasn't my problem. Got moaned at by my boss, but since it's him that put me onsite in a basement with a pager there wasn't much he could do about it.

  9. ColinPa

    No good deed goes unpunished

    I was one of a team providing on site L3 support for the Olympics. We had nothing to do, so stood around for an hour or so watching the games. We moved to the coffee machine to get a drink when our phones went crazy. 100 missed calls, texts like "What is this problem you have", "please get John to answer his phone" etc. We had calls from techies all the way up to executives! No one had realized that there was a dead spot for mobiles in the room. We asked around for the problem, and no one knew of any problems. We got in the car and went to the venue.

    When we got there, there were two windows machines one was fine the other was so busy it was hard to logon. Someone had configured antivirus to run every 10 minutes instead of once a day (at midnight). We stopped anti virus and the machine woke up.

    We asked who they had called when they found the problem. The people had ignored the instructions - "any problems phone L1 support on xxxxxx". They had a list of phone numbers for all products on the machine and picked one at random - our product.

    It took us several hours to reply to all of the emails and calm down the management chain. This incident had percolated up the chain to the CEOs office and they wanted action. Our management team were good - they managed to persuade the executives that we did not need a task force to determine the root cause. We were an innocent party - we were the good guys for fixing it - not for causing it.

    Our senior manager told us "remember no good dead goes unpunished"

    1. A.P. Veening Silver badge
      Coat

      Re: No good deed goes unpunished

      Our senior manager told us "remember no good dead goes unpunished"

      And nothing but good about the dead.

  10. Saruman the White Silver badge

    Many years ago ...

    ... when I was the IT manager for my company (not have gratefully passed that poisoned chalice to someone else) I had been configuring a new Linux server which was being used to host a couple of fairly important applications - e-mail and a Document Management System. Before going off on a nice two week holiday to Canada, I had been tweaking a couple of settings relating to the LDAP directory, and had accidentally left it in a state where the server would not boot unless it couple first contact the LDAP system, which of course was also hosted on the server. Was not able to test the settings to find the problem since we only had a production system at that time; my company was too small to afford a development/test server.

    Anyway one fine sunny morning I & my family was just visiting the Montreal Olympic Stadium when my mobile went off. It was my Director who explained that they had had a power glitch that had forced the server to shutdown, and it was now not rebooting - could I please help them sort it out. This was definitely my personal "Oh Shit ..." moment! For the next 90 minutes, while wife & children went off to see what they could see, I was stuck outside the stadium diagnosing the problem, realising what I had done, and then having to remotely guide my boss (who I should add is a pretty good guy - have known him for over 30 years) on who to boot the system into run level 1, mount the file systems, edit the faulty configuration file, and then elevate the system to run level 3.

    After that I persuaded my boss to purchase a small development/test server; I was also a lot more careful when editing configuration files after that!

    1. PC Paul

      Re: Many years ago ...

      > we only had a production system at that time; my company was too small to afford a development/test server.

      I think you'll find you did have a dev/test server, but some fool was running production on it too.

  11. Anonymous Coward
    Anonymous Coward

    It's always DNS

    Me to my boss the other day: "Hey, I was just in the server room and $OLDMACHINE is making a bad noise, I think it's on it's last legs. Is there anything still running on it?"

    Him, half an hour later: "I checked and X, Y and Z are all moved off, but the only thing still in use is a couple of older machines using it for DNS".

    It's always DNS.

    1. John Brown (no body) Silver badge
      Facepalm

      Re: It's always DNS

      "It's always DNS."

      On a similar note, I got sent to a remote office to find out why the printer was not on the network after a power cut. It was still configured to get it's network config from the BootP server that had been decommissioned about 5 years earlier.

      1. Anonymous Coward
        Anonymous Coward

        Re: It's always DNS

        At an ISP I was working for a few years ago, we had a major customer-facing outage on New Year's Day. Turns out our nicely clustered DHCP servers all filled their disc space with logs - simultaneously - so all failed at exactly the same time! As a quick fix I changed the log rotation from weekly to daily, so compression could kick in before it got insanely big.

        (When I left there, the logs are offloaded to a log server, as they should always be.)

    2. Build Monkey

      Re: It's always DNS

      And a few scats....

    3. heyrick Silver badge

      Re: It's always DNS

      Upgraded from a Livebox 2 to a Livebox Play. Got an extra half megabit on a long run of twisted pair that is older than I am and my hair is grey (I live very rural).

      Randomly, my phone (S9) is unable to talk to any of the machines on the LAN using IP address. No idea why. But since the printer has WiFi Direct (which is usually faster) I've not bothered looking into it.

      So it's not always DNS. Sometimes it's just shitty firmware.

    4. Donn Bly

      Re: It's always DNS

      When it is making noise it isn't much of a problem, it is when it STOPS making noise that I worry.

  12. big_D Silver badge

    IT's always DNS

    I had a problem a few weeks back, I went on holiday and powered everything down. When I came back home, I powered everything back up, only the Pi-Hole's DNS resolver wouldn't kick into life. The Pi-Hole software and the local network were available, but anything external failed.

    Restarting the service didn't work, restarting the Pi didn't work. In the end, I changed the DNS provider and it suddenly started working again, changed it back to the original provider and it continued working... The wonders of DNS at work.

    1. jake Silver badge

      Re: IT's always DNS

      "The wonders of DNS at work."

      Or in your case, the wonders of DNS at home.

  13. hmv

    "decided modern one"

    BIND is decidedly modern? Since when?

    It wasn't exactly a spring chicken when I started poking at it in the early 1990s.

    1. jake Silver badge

      Re: "decided modern one"

      BIND might be younger than you think. It was released to the GreatUnwashed in '86. Some still used Jon's list for a few years after that ...

      1. Pascal Monett Silver badge

        Re: It was released to the GreatUnwashed in '86

        86 ? That's practically prehistory in Computer Time.

        1. jake Silver badge

          Re: It was released to the GreatUnwashed in '86

          Don't be daft. Computers had been in wide-spread use for thirty years at that point.

  14. Kubla Cant
    Angel

    GPRS? Luxury!

    It was a long time ago*, about 9pm, and the development team had been in the pub since 6pm. In walked a completely sober project manager, who said "There's a problem with $SYSTEM at $CLIENT. Can anyone help?".

    The result was a crowd of tipsy developers standing round a VT52** connected to the client site over a 1200 bps modem. The system with the problem was fairly old, and nobody there had ever worked on it. We managed to come up with a solution, but unsurprisingly I can't remember what it was. We did some of our best work when drunk. Fortunately systems were simpler in those days.

    * I can't be bothered to work out the year, but it was before mobile phones, Internet and even LANs - all the terminals were wired back to the machine room with RS232 cables. PCs just about existed, but a serious software company wouldn't have had such things.

    ** An obsolete terminal, even then. It had suffered so much screen-burn that it was hard to make out the characters.

    1. herman

      Re: GPRS? Luxury!

      Ahhh, the wonders of 1200 baud modems: Press page down five times and go get some coffee...

    2. Mike Pellatt

      Re: GPRS? Luxury!

      God, the VT52, non-ANSI predecessor to the VT100.....

      Then there was Newbury Data, who made VDUs affordable. Had a pile of them in the HENP group at Imperial, hooked up to the DecSystem 10 via a DC-10 (not the aeroplane), whose RS232 interface had been created by kludging the current loop TTY interface. Badly.

      You brought back memories there.

  15. Anonymous Coward
    Happy

    Ah, but....

    > A quick edit with vi, and a restart

    This is a very suspect story. If "Kildare" had used vi, he'd still be trying to save and exit the editor....

    1. laughthisoff

      Re: Ah, but....

      Yeah, it's amazing how long it takes to type four characters (ESC, :, w, and q).

      Downvoted, I'm afraid, because vi can (and has) saved the universe on many occasions, and still does.

      1. TimMaher Silver badge

        Re: Ah, but....

        Or even :x?

        1. Anonymous Coward
          Anonymous Coward

          Re: Ah, but....

          Or even ZZ?

      2. Anonymous Coward
        Joke

        Re: Ah, but....

        Don't ruin the gag. Honestly, these EMACS users....

        (Anything more than a smiley required to indicate a joke?)

    2. Throatwarbler Mangrove Silver badge
      Trollface

      Re: Ah, but....

      You're thinking of emacs . . .

      1. PerlyKing

        Re: You're thinking of emacs

        Some time ago, as the resident Emacs-head I was called over to another team who couldn't work out how to save and exit from Emacs (or just exit, they didn't care at that point!). I walked over, already basking in the nerd-glory that was to be mine for solving the insoluble, to be presented with Emacs running on a Prime computer. So what? Emacs is Emacs, right? Not always, it turned out. C-x, C-c did nothing. M-x did nothing. Swearing at it did nothing. I eventually had to retire, defeated.

  16. TeeCee Gold badge

    I recall my phone going off at an inopportune time and being sure I was toast, as it was sure to be noticed I was ratarsed.

    Turned out that the on-call bloke at ${site} had had his wife walk out on him that day, had buggered off to the pub at lunchtime and had never returned. He was in a worse state than I was.

    We both spoke fluent drunken idiot and resolved the issue in short order, agreeing between us to never let anyone know the unimportant details....

    1. Chris King

      "We both spoke fluent drunken idiot and resolved the issue in short order, agreeing between us to never let anyone know the unimportant details...."

      What, like finding comments like "You're my best mate, I love you" all over the files you edited ?

      1. Aussie Doc
        Pint

        "If I wasn't drunk, how would all my mates know that I love them at 3.00am?"

      2. A K Stiles
        Pint

        drunk comments

        One of the files I worked on about 10 years ago still had comments from one of my colleagues that ran something like "2000-01-01 01:15 XXX It's New Year's morning and I'm in no state to be looking at this - can't tell what the student was trying to achieve with the last update but I've poked a couple of lines and it seems to be working now. We can worry about it when we're back in during daylight"

        There were no more recent changes to that file when I was tweaking it in 2010...

  17. Doctor Syntax Silver badge

    NT on the laptop? That means that the Nokia 9000 or later would have been available - they were introduced about the same time. Waaay better way to dial in than lugging a lappy around.

    1. Sandtitz Silver badge
      Unhappy

      "That means that the Nokia 9000 or later would have been available"

      Damn right.

      Somehow it was always ony the CEO who had the latest Nokia Communicators, the minions who could have actually used it to productive work... never.

      1. Doctor Syntax Silver badge

        One of the advantages of being freelance is that you're the one who gets to make such decisions. I wouldn't say my contracts depended on it but it made life easier. One of the high points was sitting a a pub table after a day of meetings checking the logs from the previous overnight batch when one of the visiting USians shouted to the other "Look here, he's DBAing his box on his phone".

    2. Gene Cash Silver badge

      Nokia 9000

      My boss tried for the longest time (here in the US) to get someone to sell him a Nokia 9000. Nope. No such luck. He had to wait for real smartphones.

    3. David Haig

      As a long time fan of the Nokia Communicator, and owner of one of each version, have an up vote - but there was a better small roving machine for fixing problems, and with a proper serial port - a Toshiba Libretto 70T. Proper pc with a colour, if small screen, great keyboard and pcmia slot for network or floppy drive in (https://3.14.by/en/read/toshiba-libretto-50ct)

      1. JohnG

        I still have a Libretto 50 somewhere - probably in a box in the attic.

  18. Anonymous Coward
    Anonymous Coward

    always DNS

    Years back, the whole root DNS of a major global company was the only service run on a very old HP workstation in the DC.

    Yes, workstation, not server, since there never was any budget for DNS, only for business apps.

    Little did we know many sites had redirected *every* resolver to that little dude, and not bother with setting up local DNS systems, even less with the concept of secondary DNS in the resolver ...

    Then, of course, the little out of support workstation died, causing a MAJOR incident and stopping factories for a full day.

    Bizarely, mgmt then understood how important DNS were and magically, budget appeared for a replacement :)

    1. Bruce Ordway

      Re: always DNS

      >> the concept of secondary DNS in the resolver

      Hmmm... this reminds me how friendly the internet seemed back in the 90's.

      Where I know of at least two small companies that agreed to serve as each others secondary DNS.

    2. Chris King

      Re: always DNS

      "Years back, the whole root DNS of a major global company was the only service run on a very old HP workstation in the DC"

      Been there, done that. And that was back in the days when resolvers were generally open so you never knew who was using you until you turned on query logging to debug issues...

      ...like finding several thousand NTL cable modem users and a whole Far Eastern university campus forwarding their traffic through our local resolvers, which back then were a couple of desktop PC's.

  19. John Brown (no body) Silver badge

    Paging Dr Kildare

    Thumbs up for the reference!

    1. Scott 26
      Joke

      Re: Paging Dr Kildare

      I'm not known for my patients(patience)

  20. Robert Moore

    Best advice I was ever given...

    When you fix something, especially anything out of normal working hours, make sure everyone knows it. Email the whole damn company if you have to. Let them know what the problem was and a bit about what extraordinary steps you took to fix the issue. Helps a lot with getting your hard/late/early work recognized.

    Obviously use judgement when choosing to email the entire company. :)

    1. A.P. Veening Silver badge

      Re: Best advice I was ever given...

      Don't bother with the whole company, just (your part of) the IT-department, your manager (if not in the previous group) and the whole line all the way to the top with a CC: for the CIO and CISO if they aren't in line and of course HR for the hours you spent. And do round up those hours ;)

  21. Anonymous Coward
    Anonymous Coward

    DNS and PFYs

    Ugh, recipe for disaster. I recall snapping at one or two PFYs and telling them that they were not allowed to edit any BIND related files until they read the Cricket Book. At least learn how to use dig or nslookup (no, doing a tracert from a windows command prompt doesn't mean your DNS change worked, FFS!).

    Classic PFY move is making a major DNS change, then changing the TTL from something like 1 week to 24h. "But I changed the TTL!" I guess I suck at explaining what can go wrong there, because that happened multiple times...

    1. 9Rune5
      Flame

      Re: DNS and PFYs

      Classic PFY move is making a major DNS change, then changing the TTL from something like 1 week to 24h.

      I've lost count of all the times I've heard "but we cannot change the IP address, the DNS won't propagate through all of the internet in time!".

      People absolutely refused to learn how BIND worked, nor did they want to phone a friend.

      How is this solved these days anyway? Do DNS servers default to a low TTL so that the foolz out there think modifications can be made immediately, or have the PFYs finally learned their lesson?

      1. Jou (Mxyzptlk) Silver badge

        Re: DNS and PFYs

        Most hosters have a default TTL of 1h, quite some have a TTL of 15 Minutes. And some DNS-Servers out there ignore any TTL and seem to default enforce 30 seconds (Hello google, why is your DNS server always propagated faster than any other?)

    2. Chris King

      Re: DNS and PFYs

      I had to read DNS and BIND in its entirety before I was allowed to do *anything* with DNS.

      Shame it hasn't been updated for a few years, but DNS for Rocket Scientists isn't a bad resource.

      One day, I will probably have to print this all out... just so I can roll it up and beat someone over the head with it.

    3. Trixr

      Re: DNS and PFYs

      I was a PFY at one place and found that the grizzled old walrus-moustached network guys were editing BIND zone files manually on multiple servers in a very complex environment with multiple zones combined with an AD environment, and a split namespace for internal and external-facing hosts.

      There were a LOT of typos and errors due to the manual process - forgetting to increment the zone file serial number, forgetting a reverse entry for new host records, forgetting to restart bind after a change, never running named-checkzone and introducting syntax errors, etc etc, some of which affected the AD domain that was using BIND as its forwarders.

      In my previous job, I'd created a suite of Perl scripts to manage named/bind, and passed these on in the new job after one too many screw-ups. The heavy lifting was done by configuring Bind to allow dynamic updates to the zones from specified hosts/admin workstations, and then mostly using Net::DNS::Update to add standard record types in an easy interface. Under the hood, the "add-rr" script would send any updates to all the required zone masters and create PTRs (for A records) etc without manual intervention. Other scripts did some basic maintenance tasks by typing a couple of words.

      Obviously the scripts were throughly vetted and kept simple and crystal clear as to their function. Lots of comments to indicate where to add details for new name servers etc in the event of any infrastructure changes. Since 90% of the job was adding A, PTR and CNAMEs (and other basic RRs like SRVs and MXes from time to time), the scripts stopped nearly all the "fat-finger" incidents - all that had been really needed was to ensure consistency.

      Cut to last year, when I heard from a former colleague that these scripts - customised for the environment in 2005 - are still in use today, through multiple name server refreshes. Very flattering.

    4. Anonymous Coward
      Anonymous Coward

      Re: DNS and PFYs

      Classic PFY move is making a major DNS change, then changing the TTL from something like 1 week to 24h. "But I changed the TTL!" I guess I suck at explaining what can go wrong there, because that happened multiple times..

      It isn't just PFY's anymore. I just had the same conversation last week with the new "overseas resource" (supposedly an experienced senior tech) that is replacing me about changing the TTL *after* migrating a public-facing web server from on-premise to Azure.

      The week before that he didn't bother to update Internal DNS or enable Ping on a different server he was migrated to Azure and only updated external, causing Nagios to send alerts to the CIO's phone all night because it thought that the sites were down.

      "Kids" these days just don't understand DNS (or care about it)

  22. Anonymous South African Coward Bronze badge

    It's not DNS

    There's no way it's DNS

    It was DNS

    1. PerspexAvenger

      The 4th stage is "how did that ever work?"

  23. Anonymous Coward
    Anonymous Coward

    The secondaries will handle it, right ?

    With a previous hat on, I'd got DNS all sorted out - still manually editing files, but had a master list and a script that would build a new master config for our master BIND server, and update the config for remote secondary services we had with a hosting provider. Apart from the occasional issue of someone ignoring instructions (especially on how to update the zone serial number), it worked well - much better than the manual process that was in place when I started there. When I started there were over 1200 zones - by the time I'd cleaned things up we were down to about 650, I'd also written scripts to spot "dead" zones.

    So all running swimmingly for many years, and then I was made "redundant" - quotes because while it was technically redundancy, it was really a situation engineered by one person to get rid of what was left of the engineering department.

    Anyway, as soon as I left, the idiot manager responsible for getting rid of everyone with any technical skills and who thought he was ${deity}'s gift to computing set to deconstructing everything I'd built because, he didn't understand "Linux and all that s**t" and hated that I had built reliable stuff that didn't run on Windoze. So he just pulled the plug on the DNS server - working on the basis that the secondaries would keep serving everything. Now anyone who knows anything at all about running DNS already knows what happened next ! Yup, a week later, the secondaries expired the zones and they all stopped working. Oh how I chuckled to myself when one of my old colleagues told me what had happened. Best of all, one of the zones affected was their own - and when that went, so did a few customer services (some customers lost VoIP phones).

    I do wonder what lies the idiot manager told affected customers - I know for a fact that he won't have admitted any fault on his part, probably blamed it on the hosting company. Best thing is, for a few clicks, he could have made the secondaries into masters with a few clicks of their web UI had he done it before killing the master. He couldn't even power the server back on, he'd already ripped out all the networking as well.

    That was how he approached changes - do stuff and fix problems when the customers phone in and complain !

    But not my problem any more :-)

    1. A.P. Veening Silver badge

      Re: The secondaries will handle it, right ?

      I do wonder what lies the idiot manager told affected customers - I know for a fact that he won't have admitted any fault on his part, probably blamed it on the hosting company.

      He just told the affected customers it was your act of vengeance for being made redundant. Some even may have believed him.

  24. silks

    Carphone

    I remember one family holiday in the Lake District (dad had an F-Reg Peugeot 405 to date this) we spent ages driving round so that he could get a mobile signal strong enough for a reasonably clear voice signal to provide support back to engineering HQ. In those days it was an analogue carphone with the base wired into the dash and the handset (green LED backlit buttons) on a curly cord. He had the 1.9 (same engine as the Peugeot 205 GTi) which I was quite keen to get my hands on of course, but the company car insurance brought in new T&C's on the policy just weeks before I passed my driving test!

    1. John Brown (no body) Silver badge

      Re: Carphone

      " (dad had an F-Reg Peugeot 405 to date this)"

      Not really helpful unless you specify the F being at the end or the start of the reg :-)

      1. PerlyKing
        Headmaster

        Re: F-Reg

        Not really helpful unless you specify the F being at the end or the start of the reg :-)

        The Peugeot 405 was released in 1987. Your move ;-)

        1. John Brown (no body) Silver badge

          Re: F-Reg

          <topples King and walks off whistling tunelessly>

  25. Anonymous Coward
    Anonymous Coward

    Venting

    I hate customers that tell you "THE SITE IS DOWN" when the actual problem is an end user is getting an 'invalid password' message.

    1. The Oncoming Scorn Silver badge
      Facepalm

      Re: Venting

      My users get a new system & try to sign into a web portal using the old username that we retired 11 months ago.

      This is why my automated "e-mail solution" scripts add in their username into the body.

  26. Anonymous Tribble

    It's always DNS - most of the time

    Except when it is the cleaner unplugging the server so they can use the hoover...

    Fortunately I'm no longer on call, although I did get called one night about a month after I left that company. I just politely told the call operator that I no longer worked for them and asked them to note the fact. They hadn't been told, so not their fault.

    Other calls I've had were things like "Our ISDN connection is dead". Turned out that one of their other buildings had been destroyed in a fire and their management had asked for ALL of their ISDN lines to be cancelled until it could be rebuilt - including the ones in the other buildings that were still in use.

    1. jake Silver badge

      Re: It's always DNS - most of the time

      "Except when it is the cleaner unplugging the server so they can use the hoover"

      It's their job to clean the place, floor to ceiling, board room to bog, watering plants, replacing dead light bulbs & emptying the trash in their wake. The modern world wouldn't run without janitorial staff. Extending this to include the labs that evolved into computer centers in the 1950s wasn't even thought about, it just happened.

      Janitorial staff having the keys to the entire kingdom (as it were) was the norm until we in the glass room started putting our collective foot down in the late 1970s/early 1980s. It wasn't until the late 1980s that it became uncommon. By the late 1990s it was as rare as hen's teeth. The last time I witnessed a janitor coming unannounced into a data center "in the wee hours" at a place I was consulting for was 2005 ...

      1. Anonymous Coward
        Anonymous Coward

        Re: It's always DNS - most of the time

        Back when I worked for the Tax aroudn 2005ish, before my career involved computers, I discovered that one, and only one, of the cleaners held a fairly significant Security Clearance, because without it you couldn't get into the wing where we did all the tax for celebs and government employees, and no-one would stand for dirty desks in _those_ hallowed halls.

        Fun fact: if you get hired by HMRC or the like, they make you sign the National Secrets Act and give you a copy to keep. This is largely meaningless: the Act binds us all. But this way, they *know* you know what the law says (in short "don't sell people's tax records to the media/mafia, dumbass").

  27. TrumpSlurp the Troll
    Windows

    Toshiba Satellite Pro and NT4 Server?

    I had one of those.

    I was doing a lot of stuff with email at the time and had my own Exchange Server on it.

    Amazingly useful for testing and demonstrating and investigating problems at customer sites.

    I'm pretty sure that my current mobile phone has more memory, storage and CPU power. Certainly has more processors and a higher resolution screen.

    It sometimes amazes me how leading edge (bloody expensive) technology 20 or more years ago could support major workloads and now would probably struggle with a cut down Linux. If you could get enough RAM in for anything to boot.

  28. Anonymous Coward
    Anonymous Coward

    on call rota

    When both a new father and junior diagnostician for a Mainframe manufacturing company I was on the on call rota 1 week a month, in reality IO was called out 3 out of 4 weeks as the out of ours centre knew I'd always pick up the phone. At my leaving do the true reasons for my unscheduled calls were revealed. One senior colleague used to go out every Thursday on call or not and just wasn't fit to take a call after 8pm, another just didn't really want to do standby overnight and would literally wrap his pager in a jumper, out it in the washing basket and put the washing basket in the wardrobe when he went to bed (we were on call all night and our managers could see if a pager was turned off). The team consensus was 'we don't really need the money and we know you do'. This was actually true, the extra payments for out of hours calls when I wasn't on standby paid for my car payments each month

  29. TSM

    In 2013 I had long service leave due and we organised a 3-week trip to go and visit relatives in the north of the state -- basically 1 week driving up, 1 week there, and 1 week driving back. Particularly on the way up, we'd be out of mobile range most of the time. (We went up through the interior, and came down along the more populated coastline; still mostly no signal, but more frequent spots where signal could be obtained.)

    Shortly after we set off, I got a text message from one of my colleagues asking for help with something that had broken.

    By the time we lost reception, we'd gotten to the point where I'd made a tentative diagnosis and sent him the appropriate procedure. But I didn't get confirmation that it had worked until a couple of days later, when we got to a major town.

    No, it wasn't DNS - that would have been a different team and not my problem at all.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like