Boss hands dunce's cap to chap who turned off disk monitor If it's Sunday, it must be time for On-Call, our regular look at the messes Reg readers are asked to clean up at night, on weekends or at other times when they should by rights be enjoying themselves. This week, reader Oliver recalls the time when he worked among “a small but happy team of engineers” at “an IT support company …
At least he got it sorted quite easily.
I found that Samsung's "back up solution" for Android required over 560Mb free space in order to actually do the back-up of a 32Gb system! No such trivial fix existed as from the full disk I'd already freed 560Mb - it was now delete (some of) the data to be backed up, or nothing!
Fortunately I used USB-OTG to back that data up first.
Hardly "turn key" though!
... a company I worked for - All the on-call staff moaned about being called up for pointless disk alerts, such as a database filesystem being 99% used - which was correct, the database tables had been sized to fill the filesystem, though the sparse tables themselves might be empty. When I asked one of them why they didn't do something about it, such as fixing the monitoring, he replied "But then I wouldn't get my callout payments"! Due to that sort of attitude, I didn't stay there long enough to be put on-call myself.
And, speaking of sparse files, how many people does it take to make a logic bomb?
- One Systems Programmer to insist his utility directory, /u, should be first in the search $PATH, as he had a custom version of tar that could handle sparse files correctly. OK-ish so far.
- One fat-fingered sysadmin to type "ls > pg" in the /u directory, instead of "ls | pg". Still not a problem, as the "pg" file is not executable.
- One zealous sysadmin to decide everything in /u should be made executable. Oops, the garbage "pg" file is now used instead of the system utility.
Once had a manager who wanted to scrap our weekend backups because "backups create too many out of hours support calls".
The call that provoked that request was when an operator had forgotten a password. A non-call really, but the manager was incapable of grasping that.
' Once had a manager who wanted to scrap our weekend backups because "backups create too many out of hours support calls". '
Heh, how about a CEO who cant see the point of maintaining servers and backups?
His solution?...everything on USB sticks. (I shit thee not..there's a box of them waiting to be deployed)
I don't like the stuff, but I've a bowl of popcorn reserved especially for the spectacle that will be the debacle that is only a month or so away.
Yes, I do wish I was making this up
Well RS485, so just about describable as a very slow network. but, hey, it's been there since the dawn of PCs, and runs reliably 24/7. System directs plastic granules from silos to extrusion machines, controlled by an ancient computer. Remotes are just dumb relay drivers. No such luxury as remote login, so yours truly had to drive there in the middle of the night. By the time I got there the factory flour was covered in plastic granules - lethal! The operators had eventually managed a sort of shutdown and closed off all the silos manually.
Computer simply said "network failure". I went to unplug it from the computer to do a basic continuity check and immediately realised it was plugged into the wrong D9 socket. Put it back (into the shiny clean one) and everything sprang into life. Apparently one of their guys had decided to 'tidy' up the wiring. Everything just continued to obey the last command it had received, so it was a while before, one at a time the machines over-filled.
"You can set an Exchange server correctly? How? How? How?"
I think that Gene Spafford's directions pretty much cover it. "The only truly correctly configured Exchange server is one that is powered off, cast in a block of concrete and sealed in a lead-lined room with armed guards - and even then I have my doubts."
<quote>"The only truly correctly configured Exchange server is one that is powered off, cast in a block of concrete and sealed in a lead-lined room with armed guards dumped at the bottom of the Marianas Trench - and even then I have my doubts."</quote>
FTFY!!!!
For those who flunked geography: https://en.wikipedia.org/wiki/Mariana_Trench
No, not *that* classic use of rm -rf, but..
The Financial Director where I worked a bazillion years ago needed to restore from tape the (b)ought ledger files to our SCO Unix box.
Unfortunately, he restored the files to the root of the server's disk, and rather than move them into place, he restored them again - correctly - and then issued rm -rf b* from the root folder.
Things were hunky dory for a few weeks until we rebooted the server during overnight routine maintenence and it didn't come up again.
One trip to the office later and I'd discovered that /boot was missing. Two hours pass and we were OK for the morning, but I had to wake up the MD to open the main office where the master tapes for the OS were stored in the safe.
This post has been deleted by its author
We used to have a 200MB file of random data located in the root of an older, vendor specific unix, so that when the application numpties got a path wrong and filled up root filesystem, we could run a backup with a pre-script of "rm [temp file]", then rlogin quickly and kill the relevant process (This was when a full root booted you out of ssh, where the backup agent process were already spawned and could run).
or the time the trainee operator wanted to learn unix, so added his userid without a home directory, which defaulted to root /, then did a "userdel -r [userid]" to clear out his "home directory".
It was a Saturday morning; I had woken up, but was luxuriating in a warm bed, thinking about 2 whole days off to do nothing. Then my mobile rang and I answered it; big mistake.
The manager of a site some distance away was almost screaming down the phone at me to help sort them out; no access to email, no access to data etc. etc. Tried to remote in but no response. I did ask if there had been a power cut and he confirmed that there had, but everything was now back on.
In the end, I drove there, about 4 hours through some shitty traffic. When I arrived, I found out that the local electric company had been doing some work to the power transformer nearby; the site had been informed of this, but didn't think to tell anyone in IT. As a result, when power went off, it all went onto UPS; an alert was sent, but unfortunately, the router had failed before the email was transmitted due to not having been put on the UPS by the staff member that set it up. Plus the UPS had drained down so far, it shut everything down.
Even worse, when the power came back on, it caused the breaker for the UPS circuit to trip, so nothing on that line restarted. It took me about 10 minutes to identify this, restart some of the kit, then I had to hang around for a while checking everything cam back on.
After that, another nearly 4 hour drive home. All because some git put a power lead in the wrong socket
"After that, another nearly 4 hour drive home. All because some git put a power lead in the wrong socket"
I had a similar tale from telco days. It took nearly 10 years to diagnose.
An electrician wired a client's PABX into the non-essential power (shut off at night and weekends). Batteries would fail during the weekend, but when techs went onsite to try and fault find, nothing could be discovered (the socket was clearly marked as "essential power")
As time went on, the batteries would fail earlier and earlier. It was only when someone (me) was driving back from another callout late on Friday night and the client phoned in the fault that we were able to have someone onsite as the location was on the way home (normally it was 100 miles away).
Cue cursing and swearing that the charger wasn't working. Then cursing and swearing when it was realised that no power was getting to it - and a lot more cursing ans swearing on the client side when we traced the circuit back to the power distribution board and realised what had been done.
Thankfully there was a working essential-power outlet nearby (it was supposed to be non-essential, you can guess what happened now) which we were able to plug into.
The client did not appreciate the bill.
A friend told me this story a few years ago.
It was back in the days of Windows 3.1, so Windows actually didn't do very much to protect itself against abuse..
The aforementioned friend worked for a small budget software house, and they were working on a football game. They had a reputation of development cheap, low budget game. To give you an idea of the kind of money they spent on development, the soundtrack to each match (which the packaging made a big thing of it being digitised) consisted of the CEO and his PA cheering, then shouting "Palace" in one corner of his office. That sample was used whether Crystal Palace were one of the teams in the match or not.
Apparently, the CEO called my friend one morning to say his machine wouldn't start properly. When my friend got in (he couldn't drive, so had to get a bus which took a while), he booted the machine from a floppy. As far as he could work out, the boss had stored all his documents in the Windows folder on the C drive. He'd decided to have a clear out, and someone showed him the "del *.*" command, so he happily ran it on the Windows folder. Obviously he had no backups of any description.
Good call out stories here, reminds me of past call outs. Being on call was part of most past jobs. Often I was called out more than others and was usually called for the most difficult or expensive problem. Not because I liked being called out, just the opposite. I hated it and would try to fix problems over the phone and always created a workorder or trouble ticket to fix the cause so it wouldn't happen again.
Which is why I figured I got a call at 0300 to be at the airport by 0500 to be flown by helicopter to a remote site hundreds of kilometers away. I'm tired so go into auto mode, get up, get dressed, and then I realize I don't know what tools or parts to take because I don't know what the problem is. There is no one at site so I call the control centre and ask them to look at their screen to see what the problem is. They look and say it is all good, it's still there.
I ask what he means by "still there" and the operator explains there has been a threat against the site that it is going to be blown up or something but obviously it hasn't been bombed or anything as there would be alarms. Then he says not to worry there are "people" going in at 0500.
A few more calls and it turns out I was the "people" (no kidding, just me scheduled) so I call my boss and ask for authorization to take in extra tools. What tools he asks? I give him a quick list which includes my Mini-14.
That results in more phone calls but to make a long story a bit shorter yes I was being sent to a site because it had been threatened as it was felt I was the best choice and would be able to handle whatever was going on or at least report back what was required. A nice complement and probably true but a heads up would have been nice.
We did eventually get more training which included dealing with bombs and threats. That didn't prevent a nearby site being taken and workers held hostage and another interesting call but that's a whole other, very long, story.
Being a Tech doesn't always mean boring.
I used to work for a corp. that developed voicemail software.
Back then, the largest hard disks were 512 MB each, 5.25" drives that were several inches thick and gave off a lot of heat, and they required mapping of the error blocks, so we kept the printouts with the computer that housed them so we could reformat the drives if necessary.
I was in the QA lab late one night when I got a call from the CEO that the in-house voicemail server was down - would I go look at it and see what the problem was?
I went to the server room (a broom closet re-purposed for the in-house voicemail servers) and looked at the production server. I found that the person who had setup the server had stuffed the printouts of the error blocks between the hot drives and the drives had overheated and eventually started having errors. I took the server down, pulled the drives, copied as much data off them as I could onto other drives from the QA lab, and replaced the drives. I was there until well after midnight.
A lot of people lost a lot of voicemail that night. I don't recall just who had setup that server, but I sent out a company wide email explaining that while it was a good idea to keep the error block printouts with a computer, it wasn't a very good idea to block the airflow in a computer, especially in and around physically large hard drives that give off a lot of heat. We were lucky there wasn't a fire.
Same voicemail co.
Different office.
We had some kind of token ring network (IIRC) for the in-house network.
Periodically, we would arrive at work in the morning to find the network down and the network server (or something like it) rebooted.
The server was behind the receptionist counter at the entrance.
After a few months we found that the janitorial staff were unplugging the server to use the power outlet for their vacuum cleaners...
Had a call from a company who said their NetWare 286 server (the one with the grid display?) was down. I asked them to check it was switched on and they said yes - they could see the display. Got to site and noticed that what they could actually see was the grid burned into the monochrome display and the server was actually unplugged. Following the extension lead that had been plugged into the servers plug socket, the culprit was found drilling holes in an adjacent office.
I got called out to a British Rail office in Croydon who couldn't login to their Torus server. They'd had new carpets fitted over the weekend and the fitters had cut the coax cable at each floor point.
I remember a guy who used his IBM ATX with a missing drive panel as a useful ashtray.
I got a call from an angry user whose Wang Pro was making a continuous beeping noise. When I got to the office it was sitting on the pavement outside after leaving the building via a first floor window.
Once we'd grown large enough to require something more organised that "floppy-net" we installed a DEC P60 Novel Server.
Locked in the room that served as the warehouse, hardwire into a switchless socket so no chance of accidentally turning it off or unplugging it.
Feeling pretty smug about power we couldn't understand why the server kept shutting down.
Turns out it coincided with tea & toast time. The hardwired "server" power was on the same ring as the kitchen.
New ring installed, UPS purchased.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
