BOFH: Everyone deserves a little DOWNTIME Isn't it always the way that when you're hours from the office sporting only a smartphone with a dodgy data connection that something critical claps out back at work. And then something's wrong with the VPN so you have to gain access through a convoluted chain of remote desktop, ssh and telnet links just to end up with a …
Thank you, dear Vultures, for restoring BOFH to his rightful place on a Friday. Just what I needed after this week...
To the story, great stuff again, Simon! Remote management through a convoluted chain of remote desktops, SSH connections and port forwarding into the LOMs, why does that sound so familiar? And figuring which server by trying them at random until the one your want disappears. All with people calling you (with me, normally to tell me there's a problem, which I already know, but nobody has botherred to pass the damn message on).
And Gina is still on the scene... Hope the enjoy the trip to Amsterdam :)
I figured what that ip he just disabled did. It was a firewall they installed after I flooded the Enrichment Center with a deadly neurotoxin, to make me stop flooding the Enrichment Center with a deadly neurotoxin. So get comfortable while I warm up the neurotoxin emitters.
I am sure we have all experienced the 'it will be done much quicker if I am not hassled ' phone call.I am sure we have all experienced the 'it will be done much quicker if I am not hassled ' phone call.
I find the most effective reply is "do you want me to stand here and talk to you about doing some work to fix this, or shall I just get on with actually doing it?". It usually works surprisingly well, even allowing for your normal audience types.
sysadmin win.
(yes I have had to, given a faulty nic in an LB in front of 4 dns servers *sigh*)
And having a copy on the phone is total win.
Having a cop pull up behind you, parked on the shoulder while you fix things, is kinda cool, especially when she says "Thanks for pulling off the road to do that."
Oh - and the GPRS bit? -- depends on which slicing your network uses by the way.
"Well off the top of my head I'd say it was something like a 78 per cent chance."
"So 78 per cent?"
"No, LIKE 78 per cent. So it could be 7 or 8 or 78 per cent. It's hard to tell with all these calls."
made me laugh, out loud.
So glad i have a quiet afternoon ahead of me.
We had something on those lines. A dev company that was hosting one of our servers. Their monitoring server started pushing out alerts of failing kit. The engineer logged in remotely and rebooted the kit and went back to sleep.
We woke up our account manager at 8:30am to tell him our server was down, and so was their website and probably all their other clients. They had an air-con fail (they only had a single unit we found out later). No more alerts as the monitoring server was now fried!
We soon moved the box out of their hands.
I have heard the account of how a company that shall be nameless had an aircon failure one weekend which was automatically reported to the people who managed the physical hardware. So two engineers were dispatched in a hurry. On arrival, the security guard told them that they were not allowed in because they were not on his list of people to be admitted at weekends. While they argued virtually the entire server farm fried.
This was the same company who, a couple of years later, did not want us to host a service for them in the cloud "because it was unreliable" - then discovered how much work they would have to do to accommodate our software and databases on their shiny new server farm, and suddenly decided that perhaps we should cloud host for them.
As I say, I can't say who they were and where they were based, but the medical acronym NFN may give a clue.
Reading through that has given me a headache...it's too true.
my weeks sysadmin nightmare working remotely
I just spent half a week having an email back and forth with someone who insisted a system didn't work regardless of it working (and being shown to work through screen recorders and tutorials) with every variant and setup under the sun.
After finally convincing them to let me setup a remote desktop connection to their computer to see first hand, we did the run through and, surprise surprise, it worked perfectly. They just read a single (something they had written) word wrong and thought they were in the wrong place. If they had even filled in the "problem report" form provided they would have seen their own stupidity.
Love that we're back to weeklies, have a beer on me!
I on the other hand was once asked to do an evaluation to site a new factory, before China would even have been considered. As well as a couple of development zones in the UK, I did a comparison with the Bahamas. The Bahamas came out top; low taxes, cheap loans, workforce with UK educational standards, and as the products were small and light hybrid assemblies transport costs were not a problem.
I presented it, the MD looked at the numbers longingly and then said "I just can't see myself getting it past group management."
"I was going to say I don't know," I respond, "but now it's I don't know PLUS the time of this phone call, PLUS the time to navigate back to the remote desktop app."
"So you don't really have any idea?"
"I don't know exactly what's wrong yet, so no."
I guess we've all have this discussion!!
GPRS can be handy once you go to a remote site to swap out a router and use a different IP range then get into a chicken and egg situation because the router back at base needs to have it outgoing 'route' command changed at the same time. 3G/GPRS dongle things use (used ?) more GSM channels than when using a phone as a modem so gave you more bandwidth.
Good rant. Responding to the "The Internet is not working" phone call always resulted in a de-prioitisation in the dynamically reallocatable prioritisation schedule matrix.
for too many times I've been buried down the back of a machine trying to work out whats gone wrong in the el-cheapo antique machine I've been lumbered with fixing when the inevitable call comes from the management
"When will it be up and running?"
My current boss is pretty good though.... does'nt instantly fire me when I shout back that he should f*** off and die ...
All I know is that if he was doing all this stuff from his Smartphone in the US or Canada he'd have built up such a huge data bill that he'd probably be into the tens of thousands of dollars by the time he was done.
Oh, for a sensible European style system of cellular roaming...
Of course! Who else are they going to find that can make it work. I can recall calling my mother back in the U.S. from the Philippines about a problem that was driving me nuts. After reporting all the symptoms, she told me where to kick. If that fixed it, replace this component, otherwise replace that. She was dead on. Awesome Mom!
Sorry, but I don't buy this shit!!!!
BOFH's ALWAYS leave themselves a BACK DOOR if they are any good.
It can be nothing more than an old analog modem connected to a RS-232 port on a POTS line.
Years ago, I worked (according to TPTB, it might be a case of 'fucking off' rather) at a radio station. 1970's cheapo telemetry equipment often went bonkers at the most inopportune time, and if the transmitter site based remote unit lost "sight" of its studio counterpart, it was programmed to kill the transmitter, shutting down the station. That were (IIRC) the rules (according to the Federal Crucifixion Commission) in force at the time.
Getting control meant a 25 mile drive out to the transmitter site, and power cycling the remote unit to 'reset' it. The station's engineer didn't like to do that, so he created this 'black box' that connected to a POTS line. You "call" the 'black box', it answers with a tone, you punch in a (user selectable) authorization code to get a second tone. Then you could press one of the 12 buttons on the DTMF pad and send a command to open/close a relay. He had the "*" button set to cause a one second pulse to trigger a power drop relay on the remote units' power connection. Power goes out, and one second later, it comes back on. Another minute for the two ends to handshake, and all is good. The studio end was programmed to flash a light on the audio control board informing the operator that the telemetry unit had gone off line, and the operator better power cycle the remote end within 15 minutes of the light coming on, or else.....
Nothing pisses off a station owner than the prospect of DEAD AIR.
Now, our BOFH should NOT allow this disaster to be "wasted". Time to get out the spreadsheet, and con persuade the Boss into approving the necessary kit to improve resilience in such situations.
I still saw it on a Saturday...
On another note, I hate the interview that seems to occur during a crisis. How long bla bla, are you aware bla bla. Can you expedite it bla bla.
I would really to ask in the middle of the crisis just because:
How is the family? How is that 50ft boat? Did you get the BMW recall for the ass warmer? I see the school your kids attend received another award. Are you still boning your receptionist? I like 50's in non sequential order.
I actually told that to the dot Indian manager who was the fifth or sixth person to appear at my cube to inform me of the critical issue I was trying to resolve. He has since ascended to reporting to the dot Indian CIO and I was never promoted since he has blocked my manager's promo recommendation.
Still, I would say it again.
Bootnote: That same guy suggested last Wednesday that for "stability for Black Friday" a nightly refresh job not be run on the eve of BFD. It took several hours to trace the resultant crashing servers filling the logging disk to the lack of the refresh.
That's why I ride a motorbike (well, aside from it just being so much fun :) ), that's why I get as much time out in the country as I can (far away from cell towers - or so people believe ;) ), and that's why the smartest phone I'll ever own is my V3 RAZR. Next phone will be as dumb as I can get.
Generally can't be reached. If I can, nothing I can do about it anyway.
Like the icon.. Go.. Go far. Go fast :)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2023
