Are you sure there are servers in this cold, dark basement? Welcome to On-call, our getting-a-bit-more-regular look at the odd things readers experience when called out to do things at night. This week, reader Pauly tells us that “a couple of Mondays ago I was asked to go and install some temperature-monitoring equipment at a prestigious hotel in Knightsbridge”. That's the posh bit of …
How do you end up with two IT Managers stealing all of your equipment within a year?
It sounds like this company has both severe systematic problems with recruitment, and also with how they are treating employees for that to happen to the same place twice within a year. I haven't encountered that sort of theft once during my career.
This post has been deleted by its author
If I were the suspicious sort (spoiler: I am) I would wonder if it was really the IT Manager each time. Happenning twice at the same location does seem like the connection is something to do with the location. Perhaps it's not that they treat their IT Managers badly enough for them both to take identical revenge, but rather that someone else on the staff is stealing their stuff and blaming recently fired personnel.
If I were the suspicious sort (spoiler: I am)
I think that suspicion and mild paranoia are positive traits working in IT.
It doesn't fit a supplier taking equipment. They'd just take their servers and leave the cabling untouched.
I remember having a joking discussion with some colleagues when we were about to be made redundant when the big recession hit as to how much damage we could do in 2 minutes to the IT infrastructure.
1) Cutting the cables leading to the patch panel in two places. (maximum repair time for minimal time input)
2) Cutting the BT fibre for the ISDN30.
3) Taking the hotswap drives from the servers, exiting via a fire exit and depositing them in the nearby river. (as one person could fit the lot in a carrier bag)
Our conclusion was that we could inflict near total damage to our infrastructure within a minute. They'd "know" it was us, but wouldn't have been able to prove it to a standard required in a court of law. We didn't do anything, needless to say though "AC because..."
The point being, that taking the servers is slow, hard heavy lifting for a single person and not a massive blow to the business since a DR plan could recover to a new server as quickly as the equipment can be delivered and the tapes produced. Cutting patch leads is just petty but even a moderately large site could resolve problems in a few hours in the time it takes to build new servers with two people with radios, one at the board and one walking around calling back "port 1 is a computer, port 2 is a fax machine, port 3's a phone- the user says Ext 123"
It doesn't make sense. The destruction job itself was unprofessionally pathetic. Unless he did the backup tapes as well, in which case I might be inclined to rate the job slightly higher.
"It doesn't fit a supplier taking equipment. They'd just take their servers and leave the cabling untouched."
Good point. And the supplier would just hide the old servers somewhere on site. Then, when they get the call to put in new units, they just reinstall the old ones, bill the customer for the new hardware and put it (still in the box) on eBay.
Twice, by the looks of it.
"Unless he did the backup tapes as well"
You'd be surprised (or not) at the number of outfits who either don't run backups or "backup to disk" (and those who do use backups invariably don't keep enough generations because it costs too much)
A mounted disk can be scrubbed or removed in short order
RAID != backup
Replication != backup
etc etc.
"It sounds like this company has both severe systematic problems with recruitment, and also with how they are treating employees..."
I'd say the complete lack of lighting in the "server room" was a good clue that they're running on a shoestring and cutting every possible expense.
If I were the author, I'd make sure to get paid in full before any work was started.
Oh, the 'call', from a financial company (legal loan sharks)
'We need you here right now, our RAID has failed and nobody can log in'
'OK, which server?'
'We don't know, but can you come and look for us anyway, bring some disks'
'OK, no problem, I'll look and see what you've got listed and bring a few but if it's died then it might not just be a disk'
So, on arrival, I find eleven HP DL380 servers and a pair of MSA1000 storage arrays, all flashing merrily with every amber light.
And not a single hard drive in any of them.
And no backup tapes, or tape drives.
Somewhere in the region of 80 72GB Ultra 320 SCSI disks.
Turns out the 'IT' manager had got a bit miffed at being sacked and had stolen the lot.
From three different sites.
Or the other one, where the first on call guy had thought it was a good idea to 'Initialise' the EVA5000 because it wouldn't come back online and present volumes.
The fully loaded one for the prestigious client who's name I can't even hint at for legal reasons.
Or the major client who had relocated their own data centre and managed to connect the disk shelves incorrectly and overwrite the metadata on the array for 3000 plus users' Exchange store because the IT manager thought 'changed configuration' was OK to accept...
All in a day's work. Full, 24 hour day's worth of work.
Sounds like a BOFH in the fold, are they taking one of Simon's pages?
A real bastard would have done some additional, inconspicuous damage - like short the pairs together in some hard to find location. Imagine the blue wire from one cable being cross circuited to the orange-white of a different cable altogether? Try figuring that one out.
"A real bastard would have done some additional, inconspicuous damage - like short the pairs together in some hard to find location. Imagine the blue wire from one cable being cross circuited to the orange-white of a different cable altogether? Try figuring that one out."
Close. True, a real bastard would have shorted some cables. But first they'd leave a handy metal stepladder near the rack. Then they'd cut the cables at a length necessitating use of said ladder. The BOFH would arrange so that the condensate line from the air-conditioning unit is routed to deposit condensate around the comms racks (and would spill a bottle of table salt on the floor to enhance conductivity over the normal semi-distilled state). Then they'd short some wire, but none of this blue to green stuff. No, they'd make sure to short a few random cables to mains power.
Last step on the way out: disable the lift, making the use of the metal stepladder that much more inviting when compared to lugging a company fiberglass unit down the steps.
"Imagine the blue wire from one cable being cross circuited to the orange-white of a different cable altogether? "
Imagine someone banging panel pins (those are 15-30mm long 1mm diameter nails with quite sharp points) into a few cable bundles and trunks in hard to notice areas.
I can't for one second, think that anyone would take servers home with them or cut cables unless those items were owned by the support company. Other than theft, it will come under the computer misuse act. No IT person regardless of how bitter they are would go this far upon dismissal.
I suspect the kit was supplied and there is a disputed bill. If the invoice states ownership remains with the IT company until the bill is paid, then (s)he would remove the equipment. Perhaps the IT person was on a contract, and at the end of the minimum period wasn't fully paid up to date, so took the personally owned / unpaid equipment to recover losses.
No IT person regardless of how bitter they are would go this far upon dismissal.
Sometimes people do strange things, for strange reasons.
Sometimes it's apparently just greed.
Every field has its share of people who are disturbed, distraught, or jut plain jerls, including IT. In this instance, if it wasn't the leaving sysadmins, it was someone else with the access and the knowledge of what was valuable.
Not too long ago, some rag did a writeup on more examples of this stuff, and how to deal with it.
"Every field has its share of people who are disturbed, distraught, or jut plain jerls, including IT. In this instance, if it wasn't the leaving sysadmins, it was someone else with the access and the knowledge of what was valuable."
This, combined with the access they have to potentially important systems is why it's important to do what they told us to do in my degree when firing contractors and sysadmins. Put simply, if you need to fire one, disable all his or her accounts, cancel his or her access cards (assuming your building uses them) and find a couple of burly security guards to escort them off the premises. All of which happened within 20 minutes of an ex-colleague at another company being fired, and the only reason it took 20 minutes was that his line manager needed to question him first.
I have seen worse on "Gordon Ramsay's kitchen nightmares"..!!
HOW a Greek restaurant can survive for years with bad food, badly stored etc ... ??
I have seen similar badly handled IT stuff.. The boss has NO CLUE, thinks it is all 'magic' ... IT guys work long days and nights to get the system going right, and WHO get the praise, and a promotion??? NOT the IT staff, the BOSS!! >:(
and a few month later, the boss needs to 'save money' and sacks the IT guys!! Oh and the website, that was being maintained **for free** in house??? the boss decides to get 'a professional company' to do it for £5000 a month!!! :O :O
This post has been deleted by its author
Yep, I'm sitting on my hands waiting for them to come back begging at this very moment.
I think the real costs of reversing the insourcing may just be being realised.
The CFO who decided this route has, of course, already moved on. I was the last of his reports in place on his arrival to be made redundant/retire. Except for his PA, of course, without whom his incompetence would have been more visible sooner.
...is two occasions of failed RAIDs within days at two unrelated small orgs.
First one was a call in to say the server wasn't working. On arrival I find two failed HDDs in a 3 disk array. Pointed out the flashing red LEDs to the client who said "Ah,I thought that might be important. There only used to be one flashing light". How long has the other LED been flashing? "Oh, I don't know. It was doing that when I started here last year".
A few days later, an urgent call to a site. Their HQ IT had identified a failed disk in a branch office 3 disk RAID. Courier will meet me there about 6pm to replace. I get there at 6pm. Only the security guy there. We wait there 'till 6:30pm, no sign of courier. HQ IT says to go in and have a look anyway please since the server appears to no longer be responding. We walk through and find someone has helpfully pulled the wrong drive out. HQ IT says go home, they fucked it. We'll deal with it during working hours and make them suffer a while. The courier still had not turned up by 7pm even with a 6pm guaranteed delivery.
This post has been deleted by its author
"...is two occasions of failed RAIDs within days at two unrelated small orgs."
i've seen three in the space of 5 hours but they weren't unrelated. Two sites had a shared boundary and mainns feed with a football oval where they had recently installed light towers. The third was in the club offices. Cue Thunderstorm...
"First one was a call in to say the server wasn't working. On arrival I find two failed HDDs in a 3 disk array. Pointed out the flashing red LEDs to the client who said "Ah,I thought that might be important. There only used to be one flashing light". How long has the other LED been flashing? "Oh, I don't know. It was doing that when I started here last year"."
Many years ago I did support for an art school. Something to do with sins in past life, I assume*. I used to do a monthly sweep fixing the minor problems that never got reported, generally because they didn't realise they were problems. As soon as I walk into the front office I can hear a piercing BEEEEP every coule of seconds. I could hear it two rooms away but the staff in the office seemed oblivious. 'when questioned they said it had been going on for days, at least one had wondered what it was, but none had thought to follow the noise and find out.
I tracked it two the server's UPS complaining about batteries that would no longer hold a charge.
* Actually it was kind of cool 'cause there there were lot's of hot "arty" girls.
@Cpt Blue Bear, art school support is not too bad other than the lack of pay. My first sysadmin job was for one in Santa Fe, NM. At some point, they had hired someone to trench between (rented) buildings for a coax cable run between the two. Unfortunately, the trench was back-filled with the same stuff that came out of the hole, gravel and dirt, instead of packing sand around the PVC conduit that the cable was pulled through. Eventually, the driveway was used by something heavy enough to push a rock through the wall of the pipe which allowed it to fill with water. One of the two buildings had been the campus radio station. The mast was right next to where the trench had been run.
Santa Fe might not get much rain, but it gets plenty of lightning. I put in reqs to have the trench re-worked or to pull fibre through the existing conduit, but the administration wouldn't pay for it (we had just been through a round of layoffs which had eliminated the other 75% of my department). I kept all the burned out boards due to lightning strikes and tacked them to the wall behind my desk. Eventually, an admin walked into my office and asked me what they were. I got approval to fix the problem after that.
I did a placement at a local TV station, they had failure on a piece of kit and a bike courier was ordered with a replacement card for a "support within x hours" delivery. They were cursing when he didn't turn up because it made doing the news difficult and they stopped cursing when they were called to say the motorbike courier had been in an accident.
2 stories.
I had to fly to a remote site because they had lost connectivity. I had queried when it happened etc. but they assured me nothing had changed. When I got there, one of the network cables from the server had been cut through; and they had just tied the two ends together in an attempt at a reef knot!
My brother set a company up with a system in the days of floppy disks and gave them several warnings about making copies of the disk so that if there was a failure, it would be possible to restore the data from a backup. This happened some 18 months later, so they asked him to go to site. On arrival, he confirmed the floppy was corrupt and asked for the latest copy. The person in charge opened a filing cabinet and produced a sheaf of A4 paper; on each one was a photocopy of the disk!
"the days of floppy disks"
Ah, floppies. Back in the days of SCO my Oldham-based client had an offshoot in London. If they lost the HD SCO had to be reinstalled from about half a million floppies - at least that's what it felt like. I quickly learned that if I had to go down to do that I also had to take a spare floppy drive. It was a reprographics office and after a while enough the toner got into the drive to ruin it. Then there was the day when the operator rung up to say the ceiling had just fallen in on his desk....
“'THAT'S THE SECOND BLOODY TIME THIS YEAR!' they exclaimed, and my heart sank.”
Just how in the hell are they treating their server admins?! That kind of damage is some very serious pissed off. I've know some unstable and outright crazy IT people, but that kind of action says "employer is an ass."
Ah, I also saw the post that the theft (or repo) may have been from other employees after the fact.
partially entailed doing ops on a room full of Vaxen. 780, 2x785, 8600, some microvax, lots of drives.
one day when changing a tape over thought 'feels a bit warm behind the 8600' so go and tell the ops manager. we go in, agree it feels a bit warm. go out and look at the aircon control panel. in those days it wasnt a nice lcd panel, it was a grey metal box with panel lights and a power switch. None of the 'unit failed' lights were on. there were 3 units running the area.
still we are coming out of winter and again a week later i'm thinking 'this is too hot, can't be right'. tell the ops manager, we do the same thing again.
then a fortnight later it was early may and the first lovely warm day of the year. I'm in a diferent building and cant seem to log on to anything in the cluster. rush over to the computer room to see all the loading doors open and people wafting air in. The aircon had failed. the console printers were going mad. as a vaxcluster everything major that happened came out on all the printers. but the heat was affecting the connection and the machines were dropping out, causing all the machines to report on the status not just what it saw, but the status that the other machines in the clusters saw as well, so paper was just churning out.
eventually they took the decision to just turn the power off as they couldnt regain control for an orderly shutdown. not good.
when the aircon engineer turned up, it turned out due to a blown fuse in the control panel, the 'unit failed' lamps were not working. to the best of his knowlege he reckoned we were working on 2 units most of the winter, one failed a month ago (when i noticed it getting warmer) and the last one failed that day when it struggled in the heat.
the aircon panel was re-wired so instead of 'unit failed' lamps we had 'unit running' lamps. If the lamp was off we had a problem - or a blown bulb!
thing to be learned is if you've got a hunch there is a problem, don't trust your instrumentation.
Fail safe systems only fail when they fail to fail safe.
There's an anecdote to the effect that the UK importer of De Tomaso cars employed a related tactic to deal with the reported overheating of the cars in UK traffic - he removed the bulbs from the dashboard warning lights, and owners stopped coming back to complain about the warning light.
One day I happened to be in the computer room that housed a similar-sized pile of systems: a VAXcluster, some standalone VAXes, MicroVAXes and a pair of PDP11's, all with their disks and tapes, when the aircon failed. The first thing I noticed was the computer room sound changing, with a low rumble now being absent. About three seconds later I realised that "FSCK! The aircon's gone out!", followed by a dash out the door to the sysadmin pen. Then back to the console room with four sysadmins in tow. The head honcho decided that "this, this and that machine HAVE to stay up, that comms rack too, shut down the rest, then power down all the disks except those, those and that other one. You (pointing to two colleagues) round up all the fans from the offices and bring them here. Keep the computer room doors open. GO!"
We managed to shut the lot down before any of the overtemp safeties triggered.
Given the multitude of RA81's and 82's present I expected at least a dozen casualties there, but over the next two weeks we only had three failed HDAs (70-18491-01), and one microprocessor module a couple of weeks later.
I work for a telco as a Field Engineer. I got called to a shared office block in Bristol to investigate why our rack in the basement was off the air. The basement had 2 feet of shit floating around, and it had taken out our power and battery backup.
The customer wanted me to hose it down and re-power it within 2 hours!
Steal hard drives, others pull the disks, tap them on the floor and plug them back in.
Some where (higher ed) I worked really pissed me off, before I quit I installed a hidden web server with jokes about the management, waste of resources/fraud etc and let everyone know the IP, it amused the in-mates including new staff who were introduced to the pages. That's the worst I would ever do.
A few years ago I was working at an SME in Shoreditch, which was, back then, an area notorious for its poor power supply and frequent blackouts...
We had just created a brand new infrastructure, and apart from mail (hosted by Google Apps), everything was hosted on-site in the same office as where the employees were working.
One day I had a call - everything was offline! So I rushed in, and the situation was very odd...
Seemed that there were two power feeds coming into the building, and the consequence of this was that there was power to the comms room (thankfully), however our actual office had no electricity whatsoever (was on a faulty EDF feed), and according to the building management there was no way of routing the other feed into our office (it was a managed office). Subsequently, we (~40 staff) had to set up camp in a shared meeting room which did have power, since EDF told us that the issue wouldn't be resolved for 24-48 hours! The issues here were: we didn't have Wi-Fi, there were only a few ethernet ports in the meeting room, there were about 8 mains sockets AND... Only around 10-15 staff had laptops!
All we could do was patch our core switch down to the meeting room, daisy-chain a ton of extension cables, and then also daisy-chain a few 4/8-port switches!
That was a really grim two days, although we were SO lucky that the servers still had power.
I once had the interesting sight of a building with two feeds....that were on different phases.
Somehow one PC ended up with the monitor on one phase and the base unit on the other.
Ended up with 400 volts going down the VGA lead, with PC and monitor both becoming "live".
Bang! Imploded monitor, and burnt out motherboard and lots of other damage,including several sets of brown underwear
Rather urgent survey of the rest of the site was carried out and several other buildings were found to have the same problem. Once we'd cured the problem and put each building on a single phase we ended up with around 15% reduction in electricity used
I've recently had the pleasure of making a trip to the US on the clients dime because THEIR global support team was having troubles with the submodule we make for them. By the time we actually got their (2 hour car journey, 9 hour plane ride and another 4 hours by car again) the team had decided waiting for us was going to take too long (What with the customer breathing down their neck and 5 to 6 figure losses for each day of standing around) and that maybe they should just use those service instructions they DID have. When we actually got there all the physical swapping work had been done, leaving me to just do the recovery and some basic testing and qualification. I spent the next few days in some safetymeetings (about those qualifications I or anyone else wasn't strictly licensed to do in the US and what would be needed to BE qualified), doing some sightseeing, giving some training on recovery techniques and going to a gunrange. Best trip I've had in a long time, and it didn't cost me a dime! (Except for the gunrange, that got a bit expensive... but FUN!)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
