It's always DNS, especially when you're on holiday with nothing but a phone on GPRS Did your phone ring over the New Year? No? Then spare a thought for those unfortunates who remain answerable day or night to the dread trill of a panicked On Call. Cast your mind back to a time before the woes of today, when "Brexit" was a noise one might have made after a good night out and a few bad pints, Microsoft had yet …
so I'd been through the beer stage and was contemplating entering the vodka zone when the phone rang. It was someone from the Ops Bridge that I knew on a personal as well as a professional level.
"I know you're not on call, but there's no answer to the bleep and we've got a problem with {SYSTEMNAME]. Can you come in?".
"No way, I'm long past the point of driving legally. What's happening?"
A period of drink-enhanced remote diagnosis and guidance followed with a successful resolution and resumption of processing after and hour or so. I poured myself a vodka and started to watch The Word (told you I'd been drinking).
On Monday I asked my team leader, who was supposed to be carrying the bleep, what had happened. He clammed up. I asked his drinking partner if he knew what went on.
"Oh yeah, we'd gone for a quick pint or two, the pager went so [TEAMLEADER] dropped it into his pint then went and bought another round".
I learned a lot about professionalism that night. I still haven't really forgiven you, Brian.
Back in the 80s I was doing some work with another company office. They guy I worked with offered to take me out for a pint, but as he pulled his coat on he carefully searched through all the pockets. I asked if he'd lost car keys, but he said he was just checking for a pager. Seems that the favourite game for the unlucky on-callee was to 'accidentally' drop the pager into some random coat pocket in the cloakroom as they left for the evening. If you forgot to check your pockets before leaving you could find yourself elected...
A good reason never to be the last to leave. See icon.
Welcome to On Call, the column in which the process is always followed, always professional and nothing ever goes wrong to highlight the flaws in the system.
That being said, I cannot agree with your sentiment enough and if I were in charge, I would keep a tank of Piranhas for offender's danglies to be dangled in, OSHA be damned.
On call roster - What is that. The Ops team had a list of names and numbers for the different application suites. I am sure they had my number on speed dial.
I really don't miss being on-call.
Beer for those of us who are no longer chained to the help desk.
> The Ops team had a list of names and numbers for the different application suites
That's the way it should work: the people responsible for the code are the ones that get bothered.
It's amazing how quickly the code gets fixed when that procedure is followed.
Unfortunately there's no way I can forward my alerts (nearly all of which are caused by random Oracle webservice failures, usually returning a 502 Bad Gateway, the fix being to send the exact same request a minute or two later) to the people at Oracle who wrote the webservice code. So I have to deal with the fallout.
I was engaged to be married to the most lovely Amsterdammer but was put on call in the Netherlands by an incompetent US boss. Months before the project went live and was still in test. Other end of the country so we had to live apart. I had to cycle three miles to work whenever the phone rang. Cycling for a Scot, it's like bungee jumping if you are English, it's a wee bit of a shock to the system at 3am.
Then I had to diagnose clustered NT Exchange servers, while I hadn't yet been given any Exchange course or even advice. The blue dots are doing this, the yellow dots are doing that, and are the red dots just in my eyes?
Reader, I didn't marry her. Got the fuck out of dodge.
I think I was only hired as a joke by the sardonic Dutch. He was fat as a gas giant, I was skinny as a rake, he was Denny, I was Danny. That's Dutch humour.
He eventually put us both on an Exchange course in Edinburgh, asked me to pick him up at his hotel at half seven, drive him there. Which I did, but he was furious at me when I arrived.
"I've been waiting an hour"
"You told me to come at half seven"
"But half seven is 6:30 in Dutch"
"I know, but I'm not Dutch and this isn't the Netherlands"
Try a "Dreiviertel" oder "Viertel" Time in Germany. Dreiviertel = three-quarters, Viertel = Quarter.
Dreiviertes-sechs, aka, three-quarter-six means, depending on the Region of Germany: 5:45 or 6:45, am or pm "as the mood decides".
Viertel-Sechs, aka Quarter-six can mean, depending on the region of Germany, sometimes from town to town within the region...:
5:15 (since it is already a quarter into six)
5:45 (since it is only a quarter until six, the preferred interpretation of my region (ugh I hate it))
6:15 (six and a quarter, which is my preference).
There are things in Germany I don't like, so I always use "Sechs Uhr fünfzehn" which is the only safe was to express 6:15 so every part of Germany has the same reference.
Luckily we are spared of that AM/PM Madness - why does the AM/PM notation not switch exactly at midnight and exactly at noon?
"why does the AM/PM notation not switch exactly at midnight and exactly at noon?"
It's in the name, and from the Latin. AM is ante meridiem (before midday), and PM is post meridiem (after midday). Noon and midnight aren't considered AM or PM, they are just named markers of zero duration between the two.
Try to remember, when this notation was first being developed the concept of minutes didn't exist yet, much less seconds.
After one New Years day spent trying to fix a system in London with the bit of wet string that was the internet from very rural Natal, because the person who was supposedly on call had gone walkabout and not taken the support phone with him, I made sure of two things from then on.
1) If I am not on call over Christmas/New Year, everyone knows this well before and they all see that I'm leaving the 'On Call' Phone in the capable hands of those who are.
2) I go to places where the only internet is via Satellite and as I don't have a Sat Phone... then tough.
Worked very well until a new PHB came along and demanded that everyone be on call each and everyday over the holidays. When we asked 'What about you?'. His reply was 'I'm Sailing my boat from Lymington to Faro'. That didn't go down well so no one volunteered to be on Call.
When the MD found out, our new PHB was gone, never to be seen or heard from again. Normal service was maintained and cover suppled.
Lesson: Change freeze kicks in a couple of days before people actually go away.
People may not like it, but everyone is happier in the long run.
Unless of course you have paid on-call and people are using it to top up their income. Most lack of on-call-volunteerism can be solved with cash.
A simple tactic is to tell everyone you're going to be out of contact, because you'll be spending most of your holiday exploring deep gorges/rain forests/coral reefs/etc with no reception. Many people will believe you because there is a strange belief that outside Britain and the US, mobile coverage is poor, when the situation is exactly the opposite. Then, when at your destination, turn the phone off and lock it securely in a nice steel hotel safe - just to be sure.
I can drive about 25 miles from my house to an area of probably several square miles with no cell reception from any of the major carriers (not sure about minor ones) since it is an area where a lot of Amish live (I guess they don't rent their land out for cellular towers)
If I wanted to say I was going to be out of contact, I could just say I'm going to the farmer's market there for the day I guess.
I only have to go to my Rumpus room and I get no service. In my upstairs rooms I have marginal service from one of the three major carriers in Oz. In my lounge room I get minimal service from all three carriers. If I want good service I need to go outside to near the pool...
Dead spots are everywhere here.
A friend of mine in the Dalek building community, used to work out of a aircraft hanger, that backed onto Amish owned land.
Testing his remote control\automated creations out at random intervals used to really piss them off, far more than the planes did.
Icon - He runs a distillery now, so here's a toast to his new venture.
There's a long no-signal stretch on my semi-regular rail journey to visit my dad.
No, not the coastal stretch under the cliffs and through tunnels - that's only maybe ten minutes. But on two mainlines and the stop where I change between them - only an hour and a half by intercity from London.
Ah yes. That option. Was talking to the local Parish Council about my company's plans for FTTH into the village, and the possibility of our providing some public WiFi. One member commented that the lack of mobile coverage, WiFi and (even) decent wired Internet was what attracted some people to the place.
Quick as a flash, I pointed out that there was always the "off" button. Sage nods of agreement all round the room.
Until very recently the only reception we could get for mobiles here was a 10 sq ft area outside (its always raining) a barn 50 yds from the house.Found this out when lightning took out the landline and internet and my dad got very ill. I didnt find the spot - some spotty 15yr old guest did!
Always reminded me of the joke about the woman complaining she could see her neighbour naked and when asked by the police had to climb on the wardrobe.
I would just say "my phone will be on the kitchen table, or on silent, switched off, or just won't be answered".
No need to come up with excuses when I'm on my own time! (Not quite the total bastard, if a colleague did phone and I happened to hear it, and was in a position to help, I would, but don't expect me to be prepared for such a scenario!)
My office at a minesite has no mobile reception, so I only get notifications of missed calls once I start driving to someplace else. The other fun aspect is trying to call out, basically wander up the hill out the back, and stand outside in the 40 degree heat and flies.
Also very poor reception back at the accommodation village, again you have to have your conversations outside.
All of which is a huge improvement over my early days, when you had to drive for 2 hours just to get to a pub with a payphone...
No need for any hijinks here - I do normally spend a week or so down at my dad's place over the holidays, and as they're in a semi-rural area there is indeed no mobile signal either there or on most of the trip down. (If you really desperately need signal, there are a couple of places on the property where you might be able to get a trace of it, if you wave the phone around enough and the wind is in the right direction. Had to call Microsoft for an activation code while I was there once, that was fun.)
It's always interesting heading into one of the nearby towns from their place - and suddenly receiving a whole set of increasingly frantic text messages and missed call notifications, because something broke and my colleagues couldn't figure out how to fix it.
Upper Peninsula (U.P.) of Michigan. Went snowmobiling with in-laws once, some April in the 200x decade, outside Munising. I assumed my Samsung flip phone could go a few days -- wasn't planning on making a lot of calls -- so I didn't bring my charger. Instead, it wasted power for hours trying to find signal while we were jetting around the woods, and died an hour into the ride home just as we got to the Mackinac Bridge.
I managed to find one spot in the Taquhamenon Falls State Park campground where there was a mobile signal. If I moved a metre or so in any direction - nothing at all. Checked my work email (I know, foolish move) only to find a global "reply all" incident running rampant. Filled my poor Blackberry (yes, this was some time ago) with idiotic messages such as "please remove me from this thread" also replied to all.
Still, some of the best camping, hiking, sight-seeing I've done in a long time.
I had a boss who complained he could never get me on my phone at weekends. I pointed out that I wasn't being paid for being on call and even if I was the fact he insisted Friday afternoons were serious drinking sessions - he once threatened to fire me if I didnt come down the pub where he was calling from NOW - it would have been a bit pointless.
... my Satellite Pro (a 400CDT, if I remember correctly) triple booted. 4.4BSD, Minix 1.5 and this new-fangled thing called Slackware Linux. Slack had in the last year and a half taken over from BSD as my go-to OS. I wouldn't have been caught dead troubleshooting network issues with a Redmond based program loader pseudo-OS.
Many moons ago, I bid on a contract at a un*x shop. I won the contract without a face-to-face interview. When I walked in on the first morning, the guy in charge of the data center looked startled & exclaimed "Where's your beard‽‽‽" ... Despite over forty years of un*x experience, I do not now and never have had a beard. Still makes me chuckle :-)
Now git orf me lawn! (Beers all around.)
A long time ago, everyone in my IT group had to take a turn providing overnight and weekend cover for the haulage business (all their real work is done at night when they re-stock the supermarkets). I told the head of IT that there was no mobile coverage where I lived, but he insisted. So for four years, every month there were no responses to any calls for a week, when it was my turn.
The company didn't collapse, I didn't get a bollocking, and I received the on-call supplementary payment. Everything must have just worked when I was on-call - I must have been a lucky talisman - I saw that the other guys logged lots of calls (unless that was a scam).
We had a guy who didn't have a telephone - at home. When he was on call, he'd toddle off to the telephone box 2 streets away and phone in at around 9 in the evening, before toddling back home and going to bed, if there hadn't been any problems. That was in the days before mobile telephones.
Current workplace has servicedesk people on duty 24/7.
People with knowledge are not on call. BUT, if things goes wrong, they will start calling managers, who then have to try to contact their employees. For now, there has always been someone who could help. And we are good at finding hacks to get things running until the right persons are available.
I have been guiding people over the phone from holidays in Italy. But I never bring a PC on holidays, so people can not expect more than that. And if it is work, I will usually answer only when it is convenient. So I mostly get SMS messages telling me to read mail.
BTW: After the on-call fees where suspended, all systems started to run much better during night. Now the people responsible did not want to get disturbed anymore.
$currentjob has a full shutdown 1st week in July. This year we found out that due to production backlog, we'd have limited production for that week (except for the 4th itself - USA here). Management was looking for volunteers to be available to provide support for any issues that might arise.
Despite the late notice (1 day before break started) I didn't have concrete plans to be anywhere, so I told my boss I'd be willing to help. Late afternoon on our last scheduled day I stopped by his office to ask about how the'd contact me if they had issues (no existing policy for support). Boss said that they were instructed to call him and he'd contact the appropriate engineer.
Then he gave me the closest thing to an evil grin that he can muster and warned me that he was spending the week at a campground with no electricity and minimal cell service, so I should expect he might not get any messages before the break ended.
In the late 1990s I was doing web development for a property centre in Edinburgh, run for solicitors. If you've tried to buy a house in Edinburgh, you'll know them by their 4 initials. Typically for web dev work in the 1990s, a large part of my efforts involved editing images so that (for example) maps were as clear as possible, but would still fit in a 16-colour GIF and zip down a 28k8 modem in moments. Then there was tearing all the crap that FrontPage insisted on cramming into the HTML. Finally there was the tricksy bit of hooking it into the live database of properties for sale and drawing through the schedules, photos, searching on various criteria etc etc. It was pretty groundbreaking in its day, and to be honest I'm quite proud of it (except that it didn't actually verify if there was central heating in the property, as there was a bug in that search that I only realised much later, so apologies if you bought a cold home in/around Edinburgh!)
It's all been ripped out and replaced by now, but all the while I was working there, I was in the basement of their George Street head office. There was a Seattle Coffee Company shop also in the basement, and I didn't need to surface into daylight for hours at a time. And all the while my pager was reassuringly silent. It went mad when I emerged blinking into the sunlight, mind, but that wasn't my problem. Got moaned at by my boss, but since it's him that put me onsite in a basement with a pager there wasn't much he could do about it.
I was one of a team providing on site L3 support for the Olympics. We had nothing to do, so stood around for an hour or so watching the games. We moved to the coffee machine to get a drink when our phones went crazy. 100 missed calls, texts like "What is this problem you have", "please get John to answer his phone" etc. We had calls from techies all the way up to executives! No one had realized that there was a dead spot for mobiles in the room. We asked around for the problem, and no one knew of any problems. We got in the car and went to the venue.
When we got there, there were two windows machines one was fine the other was so busy it was hard to logon. Someone had configured antivirus to run every 10 minutes instead of once a day (at midnight). We stopped anti virus and the machine woke up.
We asked who they had called when they found the problem. The people had ignored the instructions - "any problems phone L1 support on xxxxxx". They had a list of phone numbers for all products on the machine and picked one at random - our product.
It took us several hours to reply to all of the emails and calm down the management chain. This incident had percolated up the chain to the CEOs office and they wanted action. Our management team were good - they managed to persuade the executives that we did not need a task force to determine the root cause. We were an innocent party - we were the good guys for fixing it - not for causing it.
Our senior manager told us "remember no good dead goes unpunished"
... when I was the IT manager for my company (not have gratefully passed that poisoned chalice to someone else) I had been configuring a new Linux server which was being used to host a couple of fairly important applications - e-mail and a Document Management System. Before going off on a nice two week holiday to Canada, I had been tweaking a couple of settings relating to the LDAP directory, and had accidentally left it in a state where the server would not boot unless it couple first contact the LDAP system, which of course was also hosted on the server. Was not able to test the settings to find the problem since we only had a production system at that time; my company was too small to afford a development/test server.
Anyway one fine sunny morning I & my family was just visiting the Montreal Olympic Stadium when my mobile went off. It was my Director who explained that they had had a power glitch that had forced the server to shutdown, and it was now not rebooting - could I please help them sort it out. This was definitely my personal "Oh Shit ..." moment! For the next 90 minutes, while wife & children went off to see what they could see, I was stuck outside the stadium diagnosing the problem, realising what I had done, and then having to remotely guide my boss (who I should add is a pretty good guy - have known him for over 30 years) on who to boot the system into run level 1, mount the file systems, edit the faulty configuration file, and then elevate the system to run level 3.
After that I persuaded my boss to purchase a small development/test server; I was also a lot more careful when editing configuration files after that!
Me to my boss the other day: "Hey, I was just in the server room and $OLDMACHINE is making a bad noise, I think it's on it's last legs. Is there anything still running on it?"
Him, half an hour later: "I checked and X, Y and Z are all moved off, but the only thing still in use is a couple of older machines using it for DNS".
It's always DNS.
At an ISP I was working for a few years ago, we had a major customer-facing outage on New Year's Day. Turns out our nicely clustered DHCP servers all filled their disc space with logs - simultaneously - so all failed at exactly the same time! As a quick fix I changed the log rotation from weekly to daily, so compression could kick in before it got insanely big.
(When I left there, the logs are offloaded to a log server, as they should always be.)
Upgraded from a Livebox 2 to a Livebox Play. Got an extra half megabit on a long run of twisted pair that is older than I am and my hair is grey (I live very rural).
Randomly, my phone (S9) is unable to talk to any of the machines on the LAN using IP address. No idea why. But since the printer has WiFi Direct (which is usually faster) I've not bothered looking into it.
So it's not always DNS. Sometimes it's just shitty firmware.
I had a problem a few weeks back, I went on holiday and powered everything down. When I came back home, I powered everything back up, only the Pi-Hole's DNS resolver wouldn't kick into life. The Pi-Hole software and the local network were available, but anything external failed.
Restarting the service didn't work, restarting the Pi didn't work. In the end, I changed the DNS provider and it suddenly started working again, changed it back to the original provider and it continued working... The wonders of DNS at work.
It was a long time ago*, about 9pm, and the development team had been in the pub since 6pm. In walked a completely sober project manager, who said "There's a problem with $SYSTEM at $CLIENT. Can anyone help?".
The result was a crowd of tipsy developers standing round a VT52** connected to the client site over a 1200 bps modem. The system with the problem was fairly old, and nobody there had ever worked on it. We managed to come up with a solution, but unsurprisingly I can't remember what it was. We did some of our best work when drunk. Fortunately systems were simpler in those days.
* I can't be bothered to work out the year, but it was before mobile phones, Internet and even LANs - all the terminals were wired back to the machine room with RS232 cables. PCs just about existed, but a serious software company wouldn't have had such things.
** An obsolete terminal, even then. It had suffered so much screen-burn that it was hard to make out the characters.
God, the VT52, non-ANSI predecessor to the VT100.....
Then there was Newbury Data, who made VDUs affordable. Had a pile of them in the HENP group at Imperial, hooked up to the DecSystem 10 via a DC-10 (not the aeroplane), whose RS232 interface had been created by kludging the current loop TTY interface. Badly.
You brought back memories there.
Some time ago, as the resident Emacs-head I was called over to another team who couldn't work out how to save and exit from Emacs (or just exit, they didn't care at that point!). I walked over, already basking in the nerd-glory that was to be mine for solving the insoluble, to be presented with Emacs running on a Prime computer. So what? Emacs is Emacs, right? Not always, it turned out. C-x, C-c did nothing. M-x did nothing. Swearing at it did nothing. I eventually had to retire, defeated.
I recall my phone going off at an inopportune time and being sure I was toast, as it was sure to be noticed I was ratarsed.
Turned out that the on-call bloke at ${site} had had his wife walk out on him that day, had buggered off to the pub at lunchtime and had never returned. He was in a worse state than I was.
We both spoke fluent drunken idiot and resolved the issue in short order, agreeing between us to never let anyone know the unimportant details....
One of the files I worked on about 10 years ago still had comments from one of my colleagues that ran something like "2000-01-01 01:15 XXX It's New Year's morning and I'm in no state to be looking at this - can't tell what the student was trying to achieve with the last update but I've poked a couple of lines and it seems to be working now. We can worry about it when we're back in during daylight"
There were no more recent changes to that file when I was tweaking it in 2010...
One of the advantages of being freelance is that you're the one who gets to make such decisions. I wouldn't say my contracts depended on it but it made life easier. One of the high points was sitting a a pub table after a day of meetings checking the logs from the previous overnight batch when one of the visiting USians shouted to the other "Look here, he's DBAing his box on his phone".
As a long time fan of the Nokia Communicator, and owner of one of each version, have an up vote - but there was a better small roving machine for fixing problems, and with a proper serial port - a Toshiba Libretto 70T. Proper pc with a colour, if small screen, great keyboard and pcmia slot for network or floppy drive in (https://3.14.by/en/read/toshiba-libretto-50ct)
Years back, the whole root DNS of a major global company was the only service run on a very old HP workstation in the DC.
Yes, workstation, not server, since there never was any budget for DNS, only for business apps.
Little did we know many sites had redirected *every* resolver to that little dude, and not bother with setting up local DNS systems, even less with the concept of secondary DNS in the resolver ...
Then, of course, the little out of support workstation died, causing a MAJOR incident and stopping factories for a full day.
Bizarely, mgmt then understood how important DNS were and magically, budget appeared for a replacement :)
"Years back, the whole root DNS of a major global company was the only service run on a very old HP workstation in the DC"
Been there, done that. And that was back in the days when resolvers were generally open so you never knew who was using you until you turned on query logging to debug issues...
...like finding several thousand NTL cable modem users and a whole Far Eastern university campus forwarding their traffic through our local resolvers, which back then were a couple of desktop PC's.
When you fix something, especially anything out of normal working hours, make sure everyone knows it. Email the whole damn company if you have to. Let them know what the problem was and a bit about what extraordinary steps you took to fix the issue. Helps a lot with getting your hard/late/early work recognized.
Obviously use judgement when choosing to email the entire company. :)
Don't bother with the whole company, just (your part of) the IT-department, your manager (if not in the previous group) and the whole line all the way to the top with a CC: for the CIO and CISO if they aren't in line and of course HR for the hours you spent. And do round up those hours ;)
Ugh, recipe for disaster. I recall snapping at one or two PFYs and telling them that they were not allowed to edit any BIND related files until they read the Cricket Book. At least learn how to use dig or nslookup (no, doing a tracert from a windows command prompt doesn't mean your DNS change worked, FFS!).
Classic PFY move is making a major DNS change, then changing the TTL from something like 1 week to 24h. "But I changed the TTL!" I guess I suck at explaining what can go wrong there, because that happened multiple times...
Classic PFY move is making a major DNS change, then changing the TTL from something like 1 week to 24h.
I've lost count of all the times I've heard "but we cannot change the IP address, the DNS won't propagate through all of the internet in time!".
People absolutely refused to learn how BIND worked, nor did they want to phone a friend.
How is this solved these days anyway? Do DNS servers default to a low TTL so that the foolz out there think modifications can be made immediately, or have the PFYs finally learned their lesson?
I had to read DNS and BIND in its entirety before I was allowed to do *anything* with DNS.
Shame it hasn't been updated for a few years, but DNS for Rocket Scientists isn't a bad resource.
One day, I will probably have to print this all out... just so I can roll it up and beat someone over the head with it.
I was a PFY at one place and found that the grizzled old walrus-moustached network guys were editing BIND zone files manually on multiple servers in a very complex environment with multiple zones combined with an AD environment, and a split namespace for internal and external-facing hosts.
There were a LOT of typos and errors due to the manual process - forgetting to increment the zone file serial number, forgetting a reverse entry for new host records, forgetting to restart bind after a change, never running named-checkzone and introducting syntax errors, etc etc, some of which affected the AD domain that was using BIND as its forwarders.
In my previous job, I'd created a suite of Perl scripts to manage named/bind, and passed these on in the new job after one too many screw-ups. The heavy lifting was done by configuring Bind to allow dynamic updates to the zones from specified hosts/admin workstations, and then mostly using Net::DNS::Update to add standard record types in an easy interface. Under the hood, the "add-rr" script would send any updates to all the required zone masters and create PTRs (for A records) etc without manual intervention. Other scripts did some basic maintenance tasks by typing a couple of words.
Obviously the scripts were throughly vetted and kept simple and crystal clear as to their function. Lots of comments to indicate where to add details for new name servers etc in the event of any infrastructure changes. Since 90% of the job was adding A, PTR and CNAMEs (and other basic RRs like SRVs and MXes from time to time), the scripts stopped nearly all the "fat-finger" incidents - all that had been really needed was to ensure consistency.
Cut to last year, when I heard from a former colleague that these scripts - customised for the environment in 2005 - are still in use today, through multiple name server refreshes. Very flattering.
Classic PFY move is making a major DNS change, then changing the TTL from something like 1 week to 24h. "But I changed the TTL!" I guess I suck at explaining what can go wrong there, because that happened multiple times..
It isn't just PFY's anymore. I just had the same conversation last week with the new "overseas resource" (supposedly an experienced senior tech) that is replacing me about changing the TTL *after* migrating a public-facing web server from on-premise to Azure.
The week before that he didn't bother to update Internal DNS or enable Ping on a different server he was migrated to Azure and only updated external, causing Nagios to send alerts to the CIO's phone all night because it thought that the sites were down.
"Kids" these days just don't understand DNS (or care about it)
With a previous hat on, I'd got DNS all sorted out - still manually editing files, but had a master list and a script that would build a new master config for our master BIND server, and update the config for remote secondary services we had with a hosting provider. Apart from the occasional issue of someone ignoring instructions (especially on how to update the zone serial number), it worked well - much better than the manual process that was in place when I started there. When I started there were over 1200 zones - by the time I'd cleaned things up we were down to about 650, I'd also written scripts to spot "dead" zones.
So all running swimmingly for many years, and then I was made "redundant" - quotes because while it was technically redundancy, it was really a situation engineered by one person to get rid of what was left of the engineering department.
Anyway, as soon as I left, the idiot manager responsible for getting rid of everyone with any technical skills and who thought he was ${deity}'s gift to computing set to deconstructing everything I'd built because, he didn't understand "Linux and all that s**t" and hated that I had built reliable stuff that didn't run on Windoze. So he just pulled the plug on the DNS server - working on the basis that the secondaries would keep serving everything. Now anyone who knows anything at all about running DNS already knows what happened next ! Yup, a week later, the secondaries expired the zones and they all stopped working. Oh how I chuckled to myself when one of my old colleagues told me what had happened. Best of all, one of the zones affected was their own - and when that went, so did a few customer services (some customers lost VoIP phones).
I do wonder what lies the idiot manager told affected customers - I know for a fact that he won't have admitted any fault on his part, probably blamed it on the hosting company. Best thing is, for a few clicks, he could have made the secondaries into masters with a few clicks of their web UI had he done it before killing the master. He couldn't even power the server back on, he'd already ripped out all the networking as well.
That was how he approached changes - do stuff and fix problems when the customers phone in and complain !
But not my problem any more :-)
I do wonder what lies the idiot manager told affected customers - I know for a fact that he won't have admitted any fault on his part, probably blamed it on the hosting company.
He just told the affected customers it was your act of vengeance for being made redundant. Some even may have believed him.
I remember one family holiday in the Lake District (dad had an F-Reg Peugeot 405 to date this) we spent ages driving round so that he could get a mobile signal strong enough for a reasonably clear voice signal to provide support back to engineering HQ. In those days it was an analogue carphone with the base wired into the dash and the handset (green LED backlit buttons) on a curly cord. He had the 1.9 (same engine as the Peugeot 205 GTi) which I was quite keen to get my hands on of course, but the company car insurance brought in new T&C's on the policy just weeks before I passed my driving test!
Except when it is the cleaner unplugging the server so they can use the hoover...
Fortunately I'm no longer on call, although I did get called one night about a month after I left that company. I just politely told the call operator that I no longer worked for them and asked them to note the fact. They hadn't been told, so not their fault.
Other calls I've had were things like "Our ISDN connection is dead". Turned out that one of their other buildings had been destroyed in a fire and their management had asked for ALL of their ISDN lines to be cancelled until it could be rebuilt - including the ones in the other buildings that were still in use.
"Except when it is the cleaner unplugging the server so they can use the hoover"
It's their job to clean the place, floor to ceiling, board room to bog, watering plants, replacing dead light bulbs & emptying the trash in their wake. The modern world wouldn't run without janitorial staff. Extending this to include the labs that evolved into computer centers in the 1950s wasn't even thought about, it just happened.
Janitorial staff having the keys to the entire kingdom (as it were) was the norm until we in the glass room started putting our collective foot down in the late 1970s/early 1980s. It wasn't until the late 1980s that it became uncommon. By the late 1990s it was as rare as hen's teeth. The last time I witnessed a janitor coming unannounced into a data center "in the wee hours" at a place I was consulting for was 2005 ...
Back when I worked for the Tax aroudn 2005ish, before my career involved computers, I discovered that one, and only one, of the cleaners held a fairly significant Security Clearance, because without it you couldn't get into the wing where we did all the tax for celebs and government employees, and no-one would stand for dirty desks in _those_ hallowed halls.
Fun fact: if you get hired by HMRC or the like, they make you sign the National Secrets Act and give you a copy to keep. This is largely meaningless: the Act binds us all. But this way, they *know* you know what the law says (in short "don't sell people's tax records to the media/mafia, dumbass").
I had one of those.
I was doing a lot of stuff with email at the time and had my own Exchange Server on it.
Amazingly useful for testing and demonstrating and investigating problems at customer sites.
I'm pretty sure that my current mobile phone has more memory, storage and CPU power. Certainly has more processors and a higher resolution screen.
It sometimes amazes me how leading edge (bloody expensive) technology 20 or more years ago could support major workloads and now would probably struggle with a cut down Linux. If you could get enough RAM in for anything to boot.
When both a new father and junior diagnostician for a Mainframe manufacturing company I was on the on call rota 1 week a month, in reality IO was called out 3 out of 4 weeks as the out of ours centre knew I'd always pick up the phone. At my leaving do the true reasons for my unscheduled calls were revealed. One senior colleague used to go out every Thursday on call or not and just wasn't fit to take a call after 8pm, another just didn't really want to do standby overnight and would literally wrap his pager in a jumper, out it in the washing basket and put the washing basket in the wardrobe when he went to bed (we were on call all night and our managers could see if a pager was turned off). The team consensus was 'we don't really need the money and we know you do'. This was actually true, the extra payments for out of hours calls when I wasn't on standby paid for my car payments each month
In 2013 I had long service leave due and we organised a 3-week trip to go and visit relatives in the north of the state -- basically 1 week driving up, 1 week there, and 1 week driving back. Particularly on the way up, we'd be out of mobile range most of the time. (We went up through the interior, and came down along the more populated coastline; still mostly no signal, but more frequent spots where signal could be obtained.)
Shortly after we set off, I got a text message from one of my colleagues asking for help with something that had broken.
By the time we lost reception, we'd gotten to the point where I'd made a tentative diagnosis and sent him the appropriate procedure. But I didn't get confirmation that it had worked until a couple of days later, when we got to a major town.
No, it wasn't DNS - that would have been a different team and not my problem at all.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
