Techie wiped a server, nobody noticed, so a customer kept paying for six months Why hello, Monday! You beastly harbinger of another week of work and associated woes, which The Register each week welcomes with an instalment of “Who, Me?”, our reader-contributed tales of techies who make mistakes and mostly mollify their masters. This week, meet a reader who asked to be Regomized as “Sam” and once worked at …
Wow, similar situation, the application previously used a small executable on the client site. It was migrated by the vendor to an IIS(eww) server with a banal webpage. The trainer came, the users were trained. It was out of our hands. Sometime later(months?) the server was decommissioned, but not wiped!* A few days later the roars from the finance department were heard all over the office**... We, as in, me and my manager marched in, pulled up the webpage for the new banal IIS webpage, and some strange looks were exchanged between the finance department plebs(I'm sure coins were dropping). During the following days, it was discovered that during the training the cut over date wasn't really defined. So the users kept using the old system. In the end, the vendor was able to extract the months? of work from the old system, and massage it into the new system.
* One colleague wanted it for a test bed, but was too busy to pull the trigger!
**I'm pretty sure that was the day my hair started turning grey.
I feel older just talking about it!
I came in to a company who had no real understanding of there IT, I queried a server we were getting invoiced for (and why the invoices were for $0) and got told it's not a priority.
Eighteen months later I brought it up again when another invoice came across my desk, turning out the $0 is because another member had been paying it, some $80k a year, a sizeable chunk of my wage at the time, and this was an account cleared invoice (weird, but ok), so I brought it up again, how come we're paying so much for a server I had no idea about and what was on it?
Got access, found out it was a ton of personal identifiable information just publicly available to anyone who scanned it, secured it down, cleared it out and as it turned out to be for a service we'd dropped three years ago retired the service contract.
So I saved the company a fair chunk of money a year, prevented a possible data breach and untold cost that would ensue, my reward? Made redundant, turns out they don't need any IT type people...
One place we found an old Novell server (literally in a cupboard - still powered up and connected to the network) and we have zero idea what it was for.. there didn't seem to be any data stored on it and very definately no documentation so we eventually turned it off but left it in place.
Come next billing cycle, everything fell apart. The billing application sat there waiting for data that never turned up so all the bills went out with £0.00 printed on them..
Turns out the server was purely a transit server - it took data from one system and then bunged it on another - which the billing application (which I seem to remember was on some sort of mainframe) then copied it from. I suspect that the mainframe couldn't do Novell so this server pushed the data to something that the mainframe could talk to - either that or the mainframe was on a different network to the source data.
So the server got turned back on, the billing cycle was run again and this time, we sent out non-zero bills. Eventually, they re-worked and simplified things so that the server could be turned off - but that was well after I left.
SOP at the company I work for. Fire anyone with specific knowledge, some weeks later "lift and shift" migration, then some weeks later have the offshore support team start the support.
My advice to colleagues, "don't lift a finger, don't go the extra mile. The upper management arsed this up, let them wallow. The headline money saved won't reward you, but will help them earn their bonuses and the expense of your unpaid overtime". As a professional it stank, as a lesson learning exercise, it was gold.
That last paragraph is so true. My late friend would sometimes work 12-hour days, 7-days a week, and most of what he was doing was covering for other people. This or that manager wanted some ad-hoc report created, so they asked him to do it. Our mutual manager one time admitted to me that he shoved a lot of his own work onto my friend. And seemingly every year the top brass at the company made it harder and harder for anyone to get the top review rating for the biggest raise with their "stacked ranking with extra steps" system. By the time of his death (unrelated to work) he was suffering from some serious burnout.
I'll do the odd favor for individuals that I like within a company, but essentially once it's quitting time, I'm done for the day. I shut things down and they don't come back up until the following workday. The work will still be there tomorrow. And if for some reason it's not, it probably wouldn't have mattered if I did it the previous day, so it works out just the same.
Have a (virtual) pint on me.
So many people work for free, just to cover up lack of staff or mismanagement. They get nothing, but the people above them get bonuses.
I did it in the past when I was young, stupid & single. Not now.
I refuse to have email on my phone. Once I have finished for the day, it is my family time. Work is done. If something is a real emergency certain people know they can call me, but it better be a real emergency and I will be putting in the overtime request if I need to work out of hours.
Even Twitter did not invent all the mistakes.
Yes, it has invented a pile of new ones, but this kind was old before the Internet was born.
And I'm sorry, but when I'm told to unplug something, it's not my job to ensure data or function continuity. You said the Sharepoint team was now in charge and the Notes database could be wiped ?
It's wiped. Go cry next door.
I did an upgrade once, in the late 2000's that moved a system from an old version of SQL Server to a newer version. We didn't want to recreate all the DB jobs that created reports at various times and put them in an FTP folder to be picked up by other systems. With the customers chief engineer we decided that we would create those we new were used and ignore the rest until someone screamed.
Only one person said anything. We asked them what they did with the report? They e-mailed it to someone else. We talked to that person and they said they just deleted the e-mail. We went back to the first person and told them they weren't going to get that report anymore as no-one used it. We never heard anything else.
Only one person said anything. We asked them what they did with the report? They e-mailed it to someone else.
Reminds me of the "streamlining" interviews in Office Space
"What are your main job responsibilities here ? "
"What would you say you do here? throw us a bone!"
In a previous life I had a weekly report that went to a large list of folks at my company and at a client company as well.
I had a feeling it had outlived its usefulness, so for several weeks running I included a line asking anyone who wanted to continue receiving the report to let me know.
Crickets were louder than my email notifications.
I simply stopped sending it. No one asked about it for weeks. I mentioned that I had stopped sending it to my boss (who was actually a good guy) and, while he didn't quite approve of my having suspended it without explicit notice, wasn't bothered enough to ask anyone else if they missed it.
We knew no one was reading monthly report. Took a lot of resources and paper to generate and we were told oh no its very important, can do without it. So boss says put a new header on the report notifying that the report would be terminated by a date certain unless told otherwise. After a few months stopped generating the report. Never got one complaint.
I've been a bystander for the situation. Sure, telling the ex-boss to shove it gives you brief satisfaction, but having them pay you stupidly large sums of money - on your terms - is a much more satisfying experience. Yes, the pain will be felt by the shareholders. Not your problem, the gain is yours.
If your focus is on really wanting them to suffer, find some way to get past that, because that hurts you more than it hurts them. (And if "some way" means taking a baseball bat to their Beemer - which is really NOT a good idea - then for heaven's sake make sure it doesn't get caught on security video. And make sure your consulting fee greatly exceeds the fine, just in case.)
Oh, and if they try to cheap their way out of their self-made mess by offering you part payment in used lab equipment, and you happen to choose equipment that's far more valuable then the value they agree to assign to it, make sure the transactions are all complete and the gear safely carted off before you mention that fact. I'm told their facial expressions are quite a treat.
taking a baseball bat to their Beemer
Instead, smash into it with a rented garbage truck. (You removed the license plates before Beemer-smashing, then re-attached them before returning the truck, right?)
(Icon for the "Anonymous" mask you'll be wearing when you drive the truck.)
If you want to punish them, telling them to shove it is the way.
People like that are impervious to logic. You can tell them to shove it, but it simply won't register with them that your rejection was a consequence of their failure. They'll just hire someone else and (mentally) sail serenely away.
The best revenge is to make the fuckers pay you an exhorbitant rate for your (belatedly-by-them-discovered-to-be) critical knowledge.
(Icon for "posseses critical knowledge".)
《"If you want to punish them, telling them to shove it is the way." That's unprofessional. Professionals get paid.》
I would have thought in this situation the opposite of professional (like ones wife) would be amateur, as one would love (<amare) to recommend shoving a sand coated cather up his dick and piss on his serfs with that.
--
As a precaution I always like to make an archive dump of decommissioned systems before wiping them.
Years later you can always bring them up on a virtual machine or emulator.
Even if the system used time limited, now expired licenses its pretty simple to convince a VM its living in the 1990s just to extract what is required. Sort of a technological necromancy ;)
eg The non expiring Solaris Sunpro compiler flexlm licenses seemed to expire in 2000 so was a matter of revisiting 1999 to build the gcc suite under Solaris 2.4 - don't ask.
Upvoted. This is how I was taught to handle any contract job you don't want: ask for so much money either _they_ say no or if they say yes it becomes worth your time. This method may not work if the client is a lawyer.
If the motive is to have revenge on them, being overheard in the break room discussing which decisions were idiotic might have an effect.
It's not just idiocy. When actual evil is at work you stay away, regardless of the opportunity cost. The company could quite easily have hired the dev and then stiffed them at the end, or claimed that the work was not done to spec. At worst, the dev could have been sued for some trumped-up reason.
Pretty standard procedure where I work is a soft cease and a scream test. Switch the system off in such a way that it can be reactivated immediately if need be, then see if anyone complains. We've also been subjected to one. A live network link had the name of a company that we bought out and someone thought it wasn't used any more. Cue lots of tickets from customers who could no longer access the system. I wonder how many other organizations have systems from predecessors where the host or domain name was never updated.
Switch the system off in such a way that it can be reactivated immediately if need be, then see if anyone complains
We've got test servers (VMs) that are only used at year-end. The rest of the time they sit there unused (and shut down). At one point they were also used a dev servers so were up a lot more.
Fortunately, we are getting rid of that application (it uses a very old version of Oracle and fails the cyber-essentials rules) and, at that point, any last remnants of it will be gone!
I used to run a VMware host just for the reasons described in the article. I built up a server with tons of cheap storage (an array of commodity SATA drives), and loaded it with VMware. Every time a server, or even "important" peoples workstations were decommissioned, I would run a quick P-to-V conversion and store them. The VMware server rarely had any VMs actually running on it, so it didn't need tons of memory or compute. It was actually powered down most of the time. It saved the day on a couple of occasions.
.... and read by hardware that is out of support, and out of calibration such as it can't read any media other than those it wrote - sometimes. Likewise, the tapes it did write cannot be read by any other device. With a hardware encryption key that no one knows the password for or even where to plug it in.
(A long association with Mr Murphy)
Case #1: I was called to look at a "a computer." This "computer" was an IBM System/something. I asked to see the backups, and was given a box of 8" floppy diskettes which were thickly coated with dust ... on the media itself. I could see tracks etched into the media from where the dust had been ground-in by the floppy drive's read/write head. There was a red LED lit on the floppy drive (the drive was not being accessed). I asked the secretary, "How long has this light been on?" Her reply: "As long as I've worked here." The system had been in use for decades, and they had been reusing the same box of floppy diskettes for backup all that time.
Case #2: Consultants had created backup scripts for a Unix box using a 3rd-party backup program. The scripts backed up the system partitions, then the home partition, then the three application data partitions, in order ... onto /dev/st0 ... and not onto /dev/nst0 as they ought to have done. (For non-*nixers: /dev/st0 is the first tape device, and it auto-rewinds after each operation. /dev/nst0 is the first no-auto-rewind tape device. It's the same physical tape drive, though.)
Backup script from our old (BSD) Sun system was copied to our new Sun (Sys-V) system
Tape drive name changed from st0 to rst0 but nobody noticed.
Script had to be run as root
Script diligently checked the backup was readable - immediately afterwards from the same 'source'
System had / mounted on a large enough partition
User didn't notice how quickly the backup ran or that the tape wasn't making any noise
Yeah, had the bad calibration thing. A QIC-150 tape that was misbehaving, I figured out that the read head was not properly aligned so tweaked it a little, and got 1024 blocks, Tweaked it some more and skipped the first blocks to get the next 1024 blocks, ... That was a long night to get about 30MB of data.
“There are no backups”?"
LOL at this one ...
lazy stab at procedure . Not everything is backed up forever - backups for example.
Do you ever erase data? do you use new tapes everyday?
If you update the hardware in a server do you keep a backup of the data before the upgrade? forever - or just till successfully restored in the new shiny backup enabled server?
If the data/app/whatever had been migrated , as the guy was told , there *would* be backups, there would also be a working live thing.
do you use new tapes everyday?
We still use some tapes at work but we're getting rid of them (to everyones delight - we've had lots of instances where a job has apparently finished but test restores are failing and that's even after replacing the tape drives and getting new tapes).
At home, my VMs are all backed up to a local NAS which then backs itself up to a Backblaze bucket - the bucket is set to only retain data for 14 days so that the costs don't balloon.
...we had an eFax server that we paid a stupid amount of money for a support contract for. We kept saying we didn't need it (only 1 genuine fax in a year), but nope we MUST keep it as the lord's above insisted it was still needed.
Anyone the next renewal came round and we thought we'd check to see how many faxes had gone through. Zero. Not a single one. That was of course was odd, as normal we get a least 50 spam a month. Turns out the service had crashed 9 months previous.
So we powered it down, didn't renew and ripped it out 6 months later.
No one ever questioned it, except one bean counter who asked why he couldn't send a fax, and we just casually replied it had been decommissioned 2 years ago and we no longer supported faxes
This is why you:
- Retain backups of everything, long after you delete the system
- Document systems, including users
- Don't sack developers of systems for which you don't have documentation (it's literally their job to do that, and before you push their work to customers, you should be able to "do it without them").
- Put in deprecation warnings into the service before you turn it off.
- Never wipe a server. Just remove the disks. Replace with known blank disks. Re-use. And then you keep the entire original disk set. You can't keep churning on old disks anyway, so you should be replacing them, but a bunch of old disks labelled "disk 1 removed from server X" are a better backup than nothing.
As opposed to an old boss of mine whose motto was "just make it live, it'll be fine". The lack of entire test systems went down well with senior management who - as they saw it - didn't have to waste money on expensive servers they didn't need.
To my eternal surprise, this actually never caused a serious issue. Although I've often wondered whether it's related to my high blood pressure...
Indeed. I think my reaction would have been - if you haven't used it for six months, why not? You tell a little white lie - you say that no-one knew what the server was doing, as far as we were concerned it was surplus to requirements. So it was switched off for a week, and no-one shouted about it - so it was decommissioned.
The story reminded me of a long ago client, I provided remote support to them, they were running SBS 2000 at the time. Few years later the server was getting slow so popped in SBS 2003 then few years later SBS 2011. Then out of the blue, one of the directors sends me an email about a Dell whatever-it-was server. As they were in the process of being brought out so all accounts etc were being looked it.
Basically the SBS2000 was leased and the company had continued to DD the payment each month for years. I'm sure the leasing company didn't mind. No one knew and all thought it was their own kit.
Ouch!!
................that was powered on once a year for 10 days to support end of year tax calculations and reporting.
No one could be bothered to actually extract the data, even to Excel. Was easier for them to continue to pay the JDE, then Oracle, support contract than to staff and manage a data capture/migration project.
And as could be predicted by everyone here at el Reg, one year the system refused to power on. Deadder than door nails. Not even clicks or heterodyne whines from the power supplies.
Oops, too late!
And as could be predicted by everyone here at el Reg, one year the system refused to power on. Deadder than door nails. Not even clicks or heterodyne whines from the power supplies.
Oops, too late!
That is the problems with those AS/400s. They are just about immortal as long as you keep them running. Repeatedly stopping them for longer periods will kill them.
At a company where I used to work, there was a dusty old HP 3000 minicomputer in the corner of the room, not even plugged in. Apparently it contained a mirror of one of our customers' production environments and that customer was paying us a small fortune for support every year. They never called us with a problem, but we had to keep it just in case. Everybody who had any knowledge of this system had long since left the company and there was no documentation of any kind. I tried to power it up once just to see if it still worked, and it displayed a few numbers on the console and some kind of prompt. But I couldn't get any further than that. It's a good thing the customer never required any support from us, because there's no way we could have provided it. I suspect the customer had actually stopped using the system decades ago and their accounts department just kept paying the bills every year without querying it.
A few years after that, I got made redundant, along with all the other staff. The company is still in business and sending out yearly support invoices to several customers. They don't know that the office is gone, all the equipment has been sold off and the only person left on the payroll is the CEO and he has no technical skills whatsoever. I'm not sure if that counts as fraud but it certainly seems somewhat dishonest.
The lessons here are that if you decommission a system, make sure you cancel any support contracts related to it. And if you are paying a company for support, make sure you raise a ticket every now and again just to see if they can still actually provide the service you are paying for.
This vaguely reminds me of someone I worked with a while back. For all intents and purposes, she ran her department, knew the system every which way, knew where the bodies were buried, all that good stuff. She just didn't want the hassles that came with the official title of manager, so let someone else handle that part. She was getting close to retirement age and kept grousing about how every time she got close to training a replacement the company would lay them off and she'd have to start all over again. This was close to a decade ago, so I can only assume this lady has retired by now. Since I'm not overly fond of that company, I am sometimes morbidly curious about just how bad the headless chicken routine was when she left and created this huge institutional knowledge vacuum in her wake.
I was also present one time when only a single person in the entire multi-national company had bothered to maintain their login credentials for a legacy SAP system. Ended up saving the company probably an eye watering sum to have SAP do something about it, and their reward was basically a thank you from their manager. I know that person is no longer at that company, and they were the only one capable of coming up with all the ad-hoc reports people wanted, so I'm again a bit morbidly curious as to what has happened in the interim.
Down in the carpark
Where the IT-men bring the machines
And ask 'Please wipe the data'
Down in the carpark with a friend called Sherm
I was in our hut
Or was it the bar
So we deleted it just the same
Then put it on the disposal pile over there
Three weeks later, the sound of running feet
Have you got that server, we need it Monday!
You can watch the managers
now not having much fun
Now look We've got a tape machine
get the server back & restore for Monday
Ring the disposal co, HOW MUCH will we have to pay!!!!
You wouldn't believe
The things Somerset Council do
Down in the carpark
Where the chant is "fix, fix, fix"
Until the IT Kit dies smoking
Down in the carpark with friends of mine
We are not support
We are not server guys
We are just here to fix it
A different face but the BS never change
Short version,
Wipe & dispose please, this server has been powered off for months.
Three weeks later
Where the F*CK is that server, we have a consultant coming in at top $$$ on Monday to load custom software on it.
Ring disposal company to find if they still have it & can we please buy it back at whatever extortionate price the disposal people demand.
Icon - Yes I was big into Gary Numan\Tubeway Army back in the late 70's/early 80's why do you ask!
If it is only used for end of year reports and you shut it down right after the end of year freeze ends you won't notice for nearly a year. If it is needed only when you have a failover to a secondary site (and you don't conduct real honest to goodness failovers instead of just doing tabletop exercises like 99% of you do because the bosses won't permit the disruption and expense of an actual failover) you won't know you need it until you do, which could be five years down the road.
Thus why you need to take a backup of every non cookie cutter server (i.e. you don't need to do it for all 8 Exchange servers when you replace them for new Exchange servers) and basically keep it around forever. Ideally after a P2V conversion if it wasn't already a VM, so you don't have to hunt around for the right hardware to run it on...
everyone with domain knowledge is a known thing in robotic manufacturing
I did tell the story ages ago of a rival company to ours paid a bunch of highly skilled folks to build their production line, then write all the documentation, then the company fired the lot and bought in min wagers to run the line.
Which they did... for some weeks
Until they misread the documents and blew up the line.
Cue a £250 000 repair bill and the pithy comment of
"Been cheaper to retain the skilled people"
migrated an options trading system to new hardware and updated software, which took all night, ended sleeping for an hour in the boardroom before people started turning up for work. At the point when I finished, the historical data had not been moved to the new server.
Of course stupid ( me) forgot to copy the historical data across to the new server, and shut down the server. As it was options trading, the only thing the traders cared about was that their trades worked. backups only backed up the day's trades and not the historical data.
Then the excrement hit the rotating air circulation device. A high profile court case needed access to the historical records that had been wiped when the server was re-utilised. We tried getting a data recovery firm to look at the drive and see if they could recover any data but because it was on an array, no dice.
In the end the company took a hit because it could not produce the records.
I've worked at a place that did design of fiddly bits and had been for many years. This team did the design and simulation work, and all the manfacturing was shipped off elsewhere. They have a standard 20 year retention policy. The process flow was more akin to monkeys on crack generating copious piles of shit, with some nuggets of gold in there. I would constantly get called asking for more disk space, or to cleanup snapshots to make space because someone had found a 400g log file that they didn't need and deleted it, which just made things worse as we *tried* to send the data to remote DR sites, but even T-3s couldn't keep up with the load.
Anyway.... all my pleas to just do all the work on scratch volumes, and to then copy just the needed data for each step of the work flow fell on deaf ears, since they had enough money to just buy more disk space, and engineering time was expensive and it was all hands on deck to get the next batch of widgets out the door, etc. And when the work got slack, no one wanted to spend time fixing the process, they just all relaxed and took it easy until the next $FIRE came around and it was 24x7 work again.
So now people are gone, businesses closed down, and I'm still there because I'm a moron and I get tasked with pulling old data back so they can "look for a spreadsheet" or "random design document I want to reference" or "the guys who build the widgets lost the plans and do we have a copy still from 15 years ago".
So the comments on backups vs. archiving really mean alot to me.
.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
