Techie wiped a server, nobody noticed, so a customer kept paying for six months

Why hello, Monday! You beastly harbinger of another week of work and associated woes, which The Register each week welcomes with an instalment of “Who, Me?”, our reader-contributed tales of techies who make mistakes and mostly mollify their masters. This week, meet a reader who asked to be Regomized as “Sam” and once worked at …

[chivo243]

It was migrated!

Wow, similar situation, the application previously used a small executable on the client site. It was migrated by the vendor to an IIS(eww) server with a banal webpage. The trainer came, the users were trained. It was out of our hands. Sometime later(months?) the server was decommissioned, but not wiped!* A few days later the roars from the finance department were heard all over the office**... We, as in, me and my manager marched in, pulled up the webpage for the new banal IIS webpage, and some strange looks were exchanged between the finance department plebs(I'm sure coins were dropping). During the following days, it was discovered that during the training the cut over date wasn't really defined. So the users kept using the old system. In the end, the vendor was able to extract the months? of work from the old system, and massage it into the new system.

* One colleague wanted it for a test bed, but was too busy to pull the trigger!

**I'm pretty sure that was the day my hair started turning grey.

I feel older just talking about it!

[Sampler]

Re: It was migrated!

I came in to a company who had no real understanding of there IT, I queried a server we were getting invoiced for (and why the invoices were for $0) and got told it's not a priority.

Eighteen months later I brought it up again when another invoice came across my desk, turning out the $0 is because another member had been paying it, some $80k a year, a sizeable chunk of my wage at the time, and this was an account cleared invoice (weird, but ok), so I brought it up again, how come we're paying so much for a server I had no idea about and what was on it?

Got access, found out it was a ton of personal identifiable information just publicly available to anyone who scanned it, secured it down, cleared it out and as it turned out to be for a service we'd dropped three years ago retired the service contract.

So I saved the company a fair chunk of money a year, prevented a possible data breach and untold cost that would ensue, my reward? Made redundant, turns out they don't need any IT type people...

[TonyJ]

Turning off an waiting...

...is a great tactic but I doubt it would have worked given no one noticed for six months.

[Ken G]

Re: Turning off an waiting...

There are always the ones that are triggered by end of year events or audit reporting.

[Anonymous Coward]

Re: Turning off an waiting...

I saw that once.. "where is my yearly Executive report so I show it to the board". Well, it got deleted, no money for a 2000€ drive upgrade.

Anon, and huge company not named.. as I was a nuisance at the meeting I was told "no money"

[CrazyOldCatMan]

Re: Turning off an waiting...

One place we found an old Novell server (literally in a cupboard - still powered up and connected to the network) and we have zero idea what it was for.. there didn't seem to be any data stored on it and very definately no documentation so we eventually turned it off but left it in place.

Come next billing cycle, everything fell apart. The billing application sat there waiting for data that never turned up so all the bills went out with £0.00 printed on them..

Turns out the server was purely a transit server - it took data from one system and then bunged it on another - which the billing application (which I seem to remember was on some sort of mainframe) then copied it from. I suspect that the mainframe couldn't do Novell so this server pushed the data to something that the mainframe could talk to - either that or the mainframe was on a different network to the source data.

So the server got turned back on, the billing cycle was run again and this time, we sent out non-zero bills. Eventually, they re-worked and simplified things so that the server could be turned off - but that was well after I left.

[trevorde]

Obvious which company this is

"staff with important knowledge fall were let go without care"

It's Twitter, isn't it?

[Fred Daggy]

Re: Obvious which company this is

SOP at the company I work for. Fire anyone with specific knowledge, some weeks later "lift and shift" migration, then some weeks later have the offshore support team start the support.

My advice to colleagues, "don't lift a finger, don't go the extra mile. The upper management arsed this up, let them wallow. The headline money saved won't reward you, but will help them earn their bonuses and the expense of your unpaid overtime". As a professional it stank, as a lesson learning exercise, it was gold.

[aerogems]

Re: Obvious which company this is

That last paragraph is so true. My late friend would sometimes work 12-hour days, 7-days a week, and most of what he was doing was covering for other people. This or that manager wanted some ad-hoc report created, so they asked him to do it. Our mutual manager one time admitted to me that he shoved a lot of his own work onto my friend. And seemingly every year the top brass at the company made it harder and harder for anyone to get the top review rating for the biggest raise with their "stacked ranking with extra steps" system. By the time of his death (unrelated to work) he was suffering from some serious burnout.

I'll do the odd favor for individuals that I like within a company, but essentially once it's quitting time, I'm done for the day. I shut things down and they don't come back up until the following workday. The work will still be there tomorrow. And if for some reason it's not, it probably wouldn't have mattered if I did it the previous day, so it works out just the same.

Have a (virtual) pint on me.

[yetanotheraoc]

Re: Obvious which company this is

"I'll do the odd favor for individuals that I like within a company"

Yes, upvoted. There's a time to go the extra mile, and there's a time to punch out.

[Anonymous Coward]

Re: Obvious which company this is

So many people work for free, just to cover up lack of staff or mismanagement. They get nothing, but the people above them get bonuses.

I did it in the past when I was young, stupid & single. Not now.

I refuse to have email on my phone. Once I have finished for the day, it is my family time. Work is done. If something is a real emergency certain people know they can call me, but it better be a real emergency and I will be putting in the overtime request if I need to work out of hours.

[Johan Bastiaansen]

Re: Obvious which company this is

Yeah, if it's really important I'll hear it on the radio or see it on the news.

[Pascal Monett]

Re: Obvious which company this is

Even Twitter did not invent all the mistakes.

Yes, it has invented a pile of new ones, but this kind was old before the Internet was born.

And I'm sorry, but when I'm told to unplug something, it's not my job to ensure data or function continuity. You said the Sharepoint team was now in charge and the Notes database could be wiped ?

It's wiped. Go cry next door.

[Wick]

Removing resources and no-one noticed

I did an upgrade once, in the late 2000's that moved a system from an old version of SQL Server to a newer version. We didn't want to recreate all the DB jobs that created reports at various times and put them in an FTP folder to be picked up by other systems. With the customers chief engineer we decided that we would create those we new were used and ignore the rest until someone screamed.

Only one person said anything. We asked them what they did with the report? They e-mailed it to someone else. We talked to that person and they said they just deleted the e-mail. We went back to the first person and told them they weren't going to get that report anymore as no-one used it. We never heard anything else.

[jmch]

Re: Removing resources and no-one noticed

"We asked them what they did with the report? They e-mailed it to someone else. We talked to that person and they said they just deleted the e-mail."

Sounds right!!

[Uncle Slacky]

Re: Removing resources and no-one noticed

Was it a TPS report?

I think you have my stapler.

[Prst. V.Jeltz]

Re: Removing resources and no-one noticed

Only one person said anything. We asked them what they did with the report? They e-mailed it to someone else.

Reminds me of the "streamlining" interviews in Office Space

"What are your main job responsibilities here ? "

"What would you say you do here? throw us a bone!"

[Anonymous Coward]

Re: Removing resources and no-one noticed

In a previous life I had a weekly report that went to a large list of folks at my company and at a client company as well.

I had a feeling it had outlived its usefulness, so for several weeks running I included a line asking anyone who wanted to continue receiving the report to let me know.

Crickets were louder than my email notifications.

I simply stopped sending it. No one asked about it for weeks. I mentioned that I had stopped sending it to my boss (who was actually a good guy) and, while he didn't quite approve of my having suspended it without explicit notice, wasn't bothered enough to ask anyone else if they missed it.

[TheBruce]

Re: Removing resources and no-one noticed

Worked at a company of 200 employees or so. We all had to send weekly status reports to the owner. After four months I just started recycling them. Firgured he wasnt reading them all.

[TheBruce]

Re: Removing resources and no-one noticed

We knew no one was reading monthly report. Took a lot of resources and paper to generate and we were told oh no its very important, can do without it. So boss says put a new header on the report notifying that the report would be terminated by a date certain unless told otherwise. After a few months stopped generating the report. Never got one complaint.

[oiseau]

Would have done otherwise

... told the company to buzz off and that he would not help us for any amount of money.

Nah ...

I would have helped them, why not?

After all, idiots are idiots. ie: they can't help themselves.

But, being idiots, I would have asked idiot money for the fix.

.

[nintendoeats]

Re: Would have done otherwise

But remember, being able to fire people arbitrarily massages the ego of the suit who does it. Whereas, paying them out the nose 6 months later to do something just costs somebody else's money.

If you want to punish them, telling them to shove it is the way.

[Pascal Monett]

I disagree. Getting paid five times my normal hourly rate because some idiot chucked me out and couldn't be arsed to know just what his job was just makes me feel good.

I don't care whose purse it comes out of as long the money goes into mine.

[Steve Hersey]

I've been a bystander for the situation. Sure, telling the ex-boss to shove it gives you brief satisfaction, but having them pay you stupidly large sums of money - on your terms - is a much more satisfying experience. Yes, the pain will be felt by the shareholders. Not your problem, the gain is yours.

If your focus is on really wanting them to suffer, find some way to get past that, because that hurts you more than it hurts them. (And if "some way" means taking a baseball bat to their Beemer - which is really NOT a good idea - then for heaven's sake make sure it doesn't get caught on security video. And make sure your consulting fee greatly exceeds the fine, just in case.)

Oh, and if they try to cheap their way out of their self-made mess by offering you part payment in used lab equipment, and you happen to choose equipment that's far more valuable then the value they agree to assign to it, make sure the transactions are all complete and the gear safely carted off before you mention that fact. I'm told their facial expressions are quite a treat.

[Anonymous Coward]

Smashing the Beemer

taking a baseball bat to their Beemer

Instead, smash into it with a rented garbage truck. (You removed the license plates before Beemer-smashing, then re-attached them before returning the truck, right?)

(Icon for the "Anonymous" mask you'll be wearing when you drive the truck.)

[Prst. V.Jeltz]

Getting paid five times my normal hourly rate because some idiot chucked me out ...

Yeah but if its over 5 hours later* , you're still on a loser ...

*(adjust for hrs taken to do job obvs)

[An_Old_Dog]

Nope

If you want to punish them, telling them to shove it is the way.

People like that are impervious to logic. You can tell them to shove it, but it simply won't register with them that your rejection was a consequence of their failure. They'll just hire someone else and (mentally) sail serenely away.

The best revenge is to make the fuckers pay you an exhorbitant rate for your (belatedly-by-them-discovered-to-be) critical knowledge.

(Icon for "posseses critical knowledge".)

[Doctor Syntax]

Re: Would have done otherwise

"Whereas, paying them out the nose 6 months later to do something just costs somebody else's money."

Make sure the amount is above their signing level so that it's visible further up the food-chain.

[Doctor Syntax]

Re: Would have done otherwise

"If you want to punish them, telling them to shove it is the way."

That's unprofessional. Professionals get paid.

[Bebu]

Re: Would have done otherwise

《"If you want to punish them, telling them to shove it is the way." That's unprofessional. Professionals get paid.》

I would have thought in this situation the opposite of professional (like ones wife) would be amateur, as one would love (<amare) to recommend shoving a sand coated cather up his dick and piss on his serfs with that.

--

As a precaution I always like to make an archive dump of decommissioned systems before wiping them.

Years later you can always bring them up on a virtual machine or emulator.

Even if the system used time limited, now expired licenses its pretty simple to convince a VM its living in the 1990s just to extract what is required. Sort of a technological necromancy ;)

eg The non expiring Solaris Sunpro compiler flexlm licenses seemed to expire in 2000 so was a matter of revisiting 1999 to build the gcc suite under Solaris 2.4 - don't ask.

[Doctor Syntax]

Re: Would have done otherwise

Don't get mad, get even.

[A.P. Veening]

Re: Would have done otherwise

Don't get mad, get even.

And remember, some people are more even than others!

[Doctor Syntax]

Re: Would have done otherwise

Get even better.

[JWLong]

Re: Would have done otherwise

/q

Don't get mad, get even.

/q

Don't get mad, don't get even, get ahead!

[Craig 2]

Re: Would have done otherwise

A real pro asks for such an obscene amount that the company end up telling YOU to `buzz off`.... Warm glow ensues

[yetanotheraoc]

Re: Would have done otherwise

Upvoted. This is how I was taught to handle any contract job you don't want: ask for so much money either _they_ say no or if they say yes it becomes worth your time. This method may not work if the client is a lawyer.

If the motive is to have revenge on them, being overheard in the break room discussing which decisions were idiotic might have an effect.

[Fred Daggy]

Re: Would have done otherwise

That's pretty well all we talk about. We can't take enough breaks to cover all the ground.

[Antron Argaiv]

Re: Would have done otherwise

The warm glow ensues when, after telling you to "buzz off", they call you back with a contract for the obscene amount of money you named.

[Jou (Mxyzptlk)]

Re: Would have done otherwise

If you "help" them they have proof you were involved and sue you. No amount of promised money can fix that.

[Daedalus]

Re: Would have done otherwise

It's not just idiocy. When actual evil is at work you stay away, regardless of the opportunity cost. The company could quite easily have hired the dev and then stiffed them at the end, or claimed that the work was not done to spec. At worst, the dev could have been sued for some trumped-up reason.

[John Brown (no body)]

Re: Would have done otherwise

"But, being idiots, I would have asked idiot money for the fix."

He'd already been brought back once as a contractor. The fact he said no, not at any price kinda implies he not only got made redundant but then got screwed over again when contracted back in.

[Anonymous Coward]

Pretty standard procedure where I work is a soft cease and a scream test. Switch the system off in such a way that it can be reactivated immediately if need be, then see if anyone complains. We've also been subjected to one. A live network link had the name of a company that we bought out and someone thought it wasn't used any more. Cue lots of tickets from customers who could no longer access the system. I wonder how many other organizations have systems from predecessors where the host or domain name was never updated.

[Yet Another Anonymous coward]

We disconnected the fire extinguishers and nobody complained for a week so we proved fire extinguishers are unnecessary

[CrazyOldCatMan]

Switch the system off in such a way that it can be reactivated immediately if need be, then see if anyone complains

We've got test servers (VMs) that are only used at year-end. The rest of the time they sit there unused (and shut down). At one point they were also used a dev servers so were up a lot more.

Fortunately, we are getting rid of that application (it uses a very old version of Oracle and fails the cyber-essentials rules) and, at that point, any last remnants of it will be gone!

[Strahd Ivarius]

There are Oracle applications that don't fail cyber-essentials rules?

[phuzz]

Maybe, but you'll have to pay Oracle a lot of money to find out

[usbac]

I used to run a VMware host just for the reasons described in the article. I built up a server with tons of cheap storage (an array of commodity SATA drives), and loaded it with VMware. Every time a server, or even "important" peoples workstations were decommissioned, I would run a quick P-to-V conversion and store them. The VMware server rarely had any VMs actually running on it, so it didn't need tons of memory or compute. It was actually powered down most of the time. It saved the day on a couple of occasions.

[G.Y.]

Windows

That is how Windows manages its pseudo-LRU paging: declare a page not to exist; if page-fault, declare it to exist, and give it an LRU date-stamp

[Anonymous Coward]

"What do you mean when you say “There are no backups”?"

LOL at this one ...

[Ken G]

Re: "What do you mean when you say “There are no backups”?"

That's ridiculous. There should always be at least 1 backup per system. It should be over 3 years old and in a format that can't be connected to any new system.

[Fred Daggy]

Re: "What do you mean when you say “There are no backups”?"

.... and read by hardware that is out of support, and out of calibration such as it can't read any media other than those it wrote - sometimes. Likewise, the tapes it did write cannot be read by any other device. With a hardware encryption key that no one knows the password for or even where to plug it in.

(A long association with Mr Murphy)

[An_Old_Dog]

Re: "What do you mean when you say “There are no backups”?"

Case #1: I was called to look at a "a computer." This "computer" was an IBM System/something. I asked to see the backups, and was given a box of 8" floppy diskettes which were thickly coated with dust ... on the media itself. I could see tracks etched into the media from where the dust had been ground-in by the floppy drive's read/write head. There was a red LED lit on the floppy drive (the drive was not being accessed). I asked the secretary, "How long has this light been on?" Her reply: "As long as I've worked here." The system had been in use for decades, and they had been reusing the same box of floppy diskettes for backup all that time.

Case #2: Consultants had created backup scripts for a Unix box using a 3rd-party backup program. The scripts backed up the system partitions, then the home partition, then the three application data partitions, in order ... onto /dev/st0 ... and not onto /dev/nst0 as they ought to have done. (For non-*nixers: /dev/st0 is the first tape device, and it auto-rewinds after each operation. /dev/nst0 is the first no-auto-rewind tape device. It's the same physical tape drive, though.)

[Yet Another Anonymous coward]

Re: "What do you mean when you say “There are no backups”?"

Backup script from our old (BSD) Sun system was copied to our new Sun (Sys-V) system

Tape drive name changed from st0 to rst0 but nobody noticed.

Script had to be run as root

Script diligently checked the backup was readable - immediately afterwards from the same 'source'

System had / mounted on a large enough partition

User didn't notice how quickly the backup ran or that the tape wasn't making any noise

[MrBanana]

Re: "What do you mean when you say “There are no backups”?"

Yeah, had the bad calibration thing. A QIC-150 tape that was misbehaving, I figured out that the read head was not properly aligned so tweaked it a little, and got 1024 blocks, Tweaked it some more and skipped the first blocks to get the next 1024 blocks, ... That was a long night to get about 30MB of data.

[Prst. V.Jeltz]

Re: "What do you mean when you say “There are no backups”?"

“There are no backups”?"

LOL at this one ...

lazy stab at procedure . Not everything is backed up forever - backups for example.

Do you ever erase data? do you use new tapes everyday?

If you update the hardware in a server do you keep a backup of the data before the upgrade? forever - or just till successfully restored in the new shiny backup enabled server?

If the data/app/whatever had been migrated , as the guy was told , there *would* be backups, there would also be a working live thing.

[CrazyOldCatMan]

Re: "What do you mean when you say “There are no backups”?"

do you use new tapes everyday?

We still use some tapes at work but we're getting rid of them (to everyones delight - we've had lots of instances where a job has apparently finished but test restores are failing and that's even after replacing the tape drives and getting new tapes).

At home, my VMs are all backed up to a local NAS which then backs itself up to a Backblaze bucket - the bucket is set to only retain data for 14 days so that the costs don't balloon.

[l8gravely]

Re: "What do you mean when you say “There are no backups”?"

Dang, you got a fast uplink at home if you can manage to get your backups into the cloud that quickly.

[IGotOut]

Yup ..

...we had an eFax server that we paid a stupid amount of money for a support contract for. We kept saying we didn't need it (only 1 genuine fax in a year), but nope we MUST keep it as the lord's above insisted it was still needed.

Anyone the next renewal came round and we thought we'd check to see how many faxes had gone through. Zero. Not a single one. That was of course was odd, as normal we get a least 50 spam a month. Turns out the service had crashed 9 months previous.

So we powered it down, didn't renew and ripped it out 6 months later.

No one ever questioned it, except one bean counter who asked why he couldn't send a fax, and we just casually replied it had been decommissioned 2 years ago and we no longer supported faxes

[Anonymous Coward]

Re: Yup ..

I used to work in a university office which had a fax machine. The fax machine ran out of ink. Nobody noticed for six months. When they did notice, they bought more ink.

[Lee D]

This is why you:

- Retain backups of everything, long after you delete the system

- Document systems, including users

- Don't sack developers of systems for which you don't have documentation (it's literally their job to do that, and before you push their work to customers, you should be able to "do it without them").

- Put in deprecation warnings into the service before you turn it off.

- Never wipe a server. Just remove the disks. Replace with known blank disks. Re-use. And then you keep the entire original disk set. You can't keep churning on old disks anyway, so you should be replacing them, but a bunch of old disks labelled "disk 1 removed from server X" are a better backup than nothing.

[Anonymous Coward]

As opposed to an old boss of mine whose motto was "just make it live, it'll be fine". The lack of entire test systems went down well with senior management who - as they saw it - didn't have to waste money on expensive servers they didn't need.

To my eternal surprise, this actually never caused a serious issue. Although I've often wondered whether it's related to my high blood pressure...

[shah27]

I am still curious that it took 6 months for anyone to find out that this server disappeared and the app no longer worked.

[Glenn Amspaugh]

Probably some end of fiscal year reports or tax documents.

[Martin]

Indeed. I think my reaction would have been - if you haven't used it for six months, why not? You tell a little white lie - you say that no-one knew what the server was doing, as far as we were concerned it was surplus to requirements. So it was switched off for a week, and no-one shouted about it - so it was decommissioned.

[druck]

That's a bit hasty, the user might have been on holiday for a week, or two, and wouldn't have expected their equipment to be decommissioned that quickly.

[terry 1]

Sometimes things are just forgotten

The story reminded me of a long ago client, I provided remote support to them, they were running SBS 2000 at the time. Few years later the server was getting slow so popped in SBS 2003 then few years later SBS 2011. Then out of the blue, one of the directors sends me an email about a Dell whatever-it-was server. As they were in the process of being brought out so all accounts etc were being looked it.

Basically the SBS2000 was leased and the company had continued to DD the payment each month for years. I'm sure the leasing company didn't mind. No one knew and all thought it was their own kit.

Ouch!!

[spuck]

Re: Sometimes things are just forgotten

So of course, somebody in Accounts Payable and/or Property was called on the carpet for failure to audit anything, right? Oh, who am I kidding...

[heyrick]

that powered-down ‘what does it break’

How would this have helped? It took six months...

[Strahd Ivarius]

Re: that powered-down ‘what does it break’

6 months is the minimum time to go through all the steps of change management in a properly ITIL organization...

[Jou (Mxyzptlk)]

Re: that powered-down ‘what does it break’

Only the standard change with a predefined templates and tasks. Don't ask about the rest.

[Sparkus]

one had an ancient JDE instance on AS/400

................that was powered on once a year for 10 days to support end of year tax calculations and reporting.

No one could be bothered to actually extract the data, even to Excel. Was easier for them to continue to pay the JDE, then Oracle, support contract than to staff and manage a data capture/migration project.

And as could be predicted by everyone here at el Reg, one year the system refused to power on. Deadder than door nails. Not even clicks or heterodyne whines from the power supplies.

Oops, too late!

[A.P. Veening]

Re: one had an ancient JDE instance on AS/400

And as could be predicted by everyone here at el Reg, one year the system refused to power on. Deadder than door nails. Not even clicks or heterodyne whines from the power supplies.

Oops, too late!

That is the problems with those AS/400s. They are just about immortal as long as you keep them running. Repeatedly stopping them for longer periods will kill them.

[Anonymous Coward]

At a company where I used to work, there was a dusty old HP 3000 minicomputer in the corner of the room, not even plugged in. Apparently it contained a mirror of one of our customers' production environments and that customer was paying us a small fortune for support every year. They never called us with a problem, but we had to keep it just in case. Everybody who had any knowledge of this system had long since left the company and there was no documentation of any kind. I tried to power it up once just to see if it still worked, and it displayed a few numbers on the console and some kind of prompt. But I couldn't get any further than that. It's a good thing the customer never required any support from us, because there's no way we could have provided it. I suspect the customer had actually stopped using the system decades ago and their accounts department just kept paying the bills every year without querying it.

A few years after that, I got made redundant, along with all the other staff. The company is still in business and sending out yearly support invoices to several customers. They don't know that the office is gone, all the equipment has been sold off and the only person left on the payroll is the CEO and he has no technical skills whatsoever. I'm not sure if that counts as fraud but it certainly seems somewhat dishonest.

The lessons here are that if you decommission a system, make sure you cancel any support contracts related to it. And if you are paying a company for support, make sure you raise a ticket every now and again just to see if they can still actually provide the service you are paying for.

[aerogems]

This vaguely reminds me of someone I worked with a while back. For all intents and purposes, she ran her department, knew the system every which way, knew where the bodies were buried, all that good stuff. She just didn't want the hassles that came with the official title of manager, so let someone else handle that part. She was getting close to retirement age and kept grousing about how every time she got close to training a replacement the company would lay them off and she'd have to start all over again. This was close to a decade ago, so I can only assume this lady has retired by now. Since I'm not overly fond of that company, I am sometimes morbidly curious about just how bad the headless chicken routine was when she left and created this huge institutional knowledge vacuum in her wake.

I was also present one time when only a single person in the entire multi-national company had bothered to maintain their login credentials for a legacy SAP system. Ended up saving the company probably an eye watering sum to have SAP do something about it, and their reward was basically a thank you from their manager. I know that person is no longer at that company, and they were the only one capable of coming up with all the ad-hoc reports people wanted, so I'm again a bit morbidly curious as to what has happened in the interim.

[Vader]

Can't have been that important then TBH.

[BebopWeBop]

Billing

The best question of all was: “How long have we been billing clients for this?”

The normal answer is 'not long enough'.

[The Oncoming Scorn]

Down in The Car Park - To The Tune Of "Down In The Park - Tubeway Army"

Down in the carpark

Where the IT-men bring the machines

And ask 'Please wipe the data'

Down in the carpark with a friend called Sherm

I was in our hut

Or was it the bar

So we deleted it just the same

Then put it on the disposal pile over there

Three weeks later, the sound of running feet

Have you got that server, we need it Monday!

You can watch the managers

now not having much fun

Now look We've got a tape machine

get the server back & restore for Monday

Ring the disposal co, HOW MUCH will we have to pay!!!!

You wouldn't believe

The things Somerset Council do

Down in the carpark

Where the chant is "fix, fix, fix"

Until the IT Kit dies smoking

Down in the carpark with friends of mine

We are not support

We are not server guys

We are just here to fix it

A different face but the BS never change

Short version,

Wipe & dispose please, this server has been powered off for months.

Three weeks later

Where the F*CK is that server, we have a consultant coming in at top $$$ on Monday to load custom software on it.

Ring disposal company to find if they still have it & can we please buy it back at whatever extortionate price the disposal people demand.

Icon - Yes I was big into Gary Numan\Tubeway Army back in the late 70's/early 80's why do you ask!

[DS999]

You can never be sure something isn't needed

If it is only used for end of year reports and you shut it down right after the end of year freeze ends you won't notice for nearly a year. If it is needed only when you have a failover to a secondary site (and you don't conduct real honest to goodness failovers instead of just doing tabletop exercises like 99% of you do because the bosses won't permit the disruption and expense of an actual failover) you won't know you need it until you do, which could be five years down the road.

Thus why you need to take a backup of every non cookie cutter server (i.e. you don't need to do it for all 8 Exchange servers when you replace them for new Exchange servers) and basically keep it around forever. Ideally after a P2V conversion if it wasn't already a VM, so you don't have to hunt around for the right hardware to run it on...

[J. Cook]

Re: You can never be sure something isn't needed

THIS.

This chucklehead got to resurrect a pair of servers that were last backed up two years ago, because we decommisioned them and then after a year deleted them from the environment.

[Boris the Cockroach]

Sacking

everyone with domain knowledge is a known thing in robotic manufacturing

I did tell the story ages ago of a rival company to ours paid a bunch of highly skilled folks to build their production line, then write all the documentation, then the company fired the lot and bought in min wagers to run the line.

Which they did... for some weeks

Until they misread the documents and blew up the line.

Cue a £250 000 repair bill and the pithy comment of

"Been cheaper to retain the skilled people"

[Doctor Syntax]

Re: Sacking

There's a skill in writing documents to be misread but which are actually correct.

[Hazmoid]

my fault here.

migrated an options trading system to new hardware and updated software, which took all night, ended sleeping for an hour in the boardroom before people started turning up for work. At the point when I finished, the historical data had not been moved to the new server.

Of course stupid ( me) forgot to copy the historical data across to the new server, and shut down the server. As it was options trading, the only thing the traders cared about was that their trades worked. backups only backed up the day's trades and not the historical data.

Then the excrement hit the rotating air circulation device. A high profile court case needed access to the historical records that had been wiped when the server was re-utilised. We tried getting a data recovery firm to look at the drive and see if they could recover any data but because it was on an array, no dice.

In the end the company took a hit because it could not produce the records.

[streaky]

As Project Management Guru Mike Tyson Once Said..

"Everybody has got a kanban board and burn-down reports until they get punched in the face*"

Funny thing is the story sounds vaguely familiar (though I wasn't involved), wonder if I know "Sam".. :)

* He might not have said _exactly_ this.

[Bebu]

Mr Backup (W.Curtis Preston)

Looks like there is still a lot of confusion between a backup and an archive.

Recommend Curtis' "Modern Data Protection" to clarify the distinction.

I still have my copies of "Backup and Recovery" and "Unix Backup and Recovery."

[Groo The Wanderer]

Yeah, but when the "breakage" doesn't show up until six MONTHS after you've shut down the server, that isn't a particularly good way to determine if the service it provides is necessary.

[l8gravely]

Packrat archiving

I've worked at a place that did design of fiddly bits and had been for many years. This team did the design and simulation work, and all the manfacturing was shipped off elsewhere. They have a standard 20 year retention policy. The process flow was more akin to monkeys on crack generating copious piles of shit, with some nuggets of gold in there. I would constantly get called asking for more disk space, or to cleanup snapshots to make space because someone had found a 400g log file that they didn't need and deleted it, which just made things worse as we *tried* to send the data to remote DR sites, but even T-3s couldn't keep up with the load.

Anyway.... all my pleas to just do all the work on scratch volumes, and to then copy just the needed data for each step of the work flow fell on deaf ears, since they had enough money to just buy more disk space, and engineering time was expensive and it was all hands on deck to get the next batch of widgets out the door, etc. And when the work got slack, no one wanted to spend time fixing the process, they just all relaxed and took it easy until the next $FIRE came around and it was 24x7 work again.

So now people are gone, businesses closed down, and I'm still there because I'm a moron and I get tasked with pulling old data back so they can "look for a spreadsheet" or "random design document I want to reference" or "the guys who build the widgets lost the plans and do we have a copy still from 15 years ago".

So the comments on backups vs. archiving really mean alot to me.

.