I'm thinking that the person who "managed" that box before did not deserve being laid off. He deserved being fired.
The Linux box that runs the exec carpark gate is down! A chance for PostgreSQL Man to show his quality
This week's episode of On Call, as ever, comes with a warning: Be careful moving that beige box, for you may not realise what it does. Register reader "Jim" was the recipient of today's super-urgent callout, which occurred during his final week of paid employment ahead of a well-earned retirement. Jim was a 20-year veteran at …
COMMENTS
-
-
-
Friday 5th February 2021 12:46 GMT Anonymous Coward
> I'm thinking that the person who "managed" that box before did not deserve being laid off. He deserved being fired.
I think you're coming away with the wrong message from this tale. The right message, surely, is that Linux is far, far too reliable! If the gate controller had been Windows then the regular reboots needed would have soon uncovered the bad config file and it could have been fixed long before anyone needed firing. ;-)
-
Friday 5th February 2021 12:55 GMT Jenny with the Axe
You do have a point :-)
When I was a younger and stupider sysadmin, I enjoyed taunting the windows people that I didn't need to reboot my systems for every little thing. Now I have learned that regular reboots are useful, not because the OS needs them, but to find stuff like the broken config files while you've still got a reasonable chance of remembering why they were changed and what they ought to look like...
-
Friday 5th February 2021 13:48 GMT Martin Gregorie
Dunno 'bout you, but I always run a stop/start/status check after any config change on a service that is started start at boot time and expected to stay up until the next reboot. Same goes for cron jobs, like getmail, that are expected to execute every few minutes and provide a 24/7 service.
-
Friday 5th February 2021 14:52 GMT FIA
Yeah, but that's at work.
Home sysadmin can sometimes be done drunk, or with interruptions, or simply 'ah, feck it.... I'm sure it'll be fine'.... usually it is... however sometimes that 'I'm just rebooting the net to install upgrades, it'll only be 5 minutes....' can really get you in the dog house. ;)
Also, it's the interactions that get you, you reboot service a when you upgrade it, then do the same for service b, but unfortunatly that stops a starting for some reason. These are the ones that get you on a reboot.
-
Tuesday 9th February 2021 16:51 GMT Alan Brown
"Home sysadmin can sometimes be done drunk, or with interruptions"
SOHO is not enterprise
and if someone asked me to fix the bigwig carpark entrance when I only had a few days left after being made redundant I suspect they might still be waiting on my final day after "working on it" diligently
-
-
Friday 5th February 2021 14:52 GMT Cynic_999
Well sure - provided that the changes were made deliberately. It's the changes that are made inadvertently (perhaps to the wrong config file) that are not tested by re-starting the process. Or (horrors) processes you don't know about that are using the same config file as the one you do know about ...
-
-
Friday 5th February 2021 14:14 GMT Shadow Systems
At Jenny, re: config files.
Back in the days of DOS (shuddup ya young whippersnapper!) and config files that needed frequent editing, I got into the habit of making a backup copy of said files before said changes. If the edit caused something to hit the fan, replace with the previous version, restart the program, let it recover & then try to figure out what went wrong. Nine times out of ten it turned out to be a fat fingers typo that bjorked things, so fixing it wasn't difficult, but on those times when it wasn't then at least I'd had the original files to fall back upon while I researched the hell out of WTF just went kaboom. Surely the same "backup first then edit" routine would work in *nix?
And on another note, please enjoy a pint for your choice of pseudonym. For some odd reason I've pictured the classic "cute little girl with pigtails, dimples, & a red & white checked dress" carrying around a massive two handed GreatAxe like others might cuddle a teddy bear. It makes me wish I were the HR manager so I could hire you on the spot as my new BOFH trainee. =-D
*Ducks the swinging axe*
Cheers!
-
Friday 5th February 2021 16:09 GMT vincent himpe
Re: At Jenny, re: config files.
Back in the good old dos days the boards also had jumpers to do the address mapping. And the config files needed to match the settings. That kept the riffraff out of the mashinene nicht gemacht fur das gefingerpoken un mittengrabben. Der dumbkopfen mussen keepen their hands in their pockets.
Those were the days a sysadmin always carried a boot floppy and a screwdriver. Now all they do is sit behind their screen coffee slurping while remoting into a machine somewhere halfway the globe.
-
Friday 5th February 2021 18:20 GMT Anonymous Coward
Re: At Jenny, re: config files.
In my bash shell days, I had a script (or shell alias, I forget) for that. Was called simply "cma". "cma $filename" would copy the file to ~/cma/$filename. Just calling the script with no args would copy the entire directory (useful for stuff like apache).
The script did indeed cover my ass several times.
Nowadays I use svn checkins in the same manner.
-
Saturday 6th February 2021 11:40 GMT Anonymous Coward
Re: At Jenny, re: config files.
I used sccs back in the late 80s then rcs then cvs then pcs, then source un safe, then starteam finally git to keep track of changes... Must admit when it comes to huge amounts of data a directory copy is the way to go preferably to a different machine. Hated the middle 3 tools...
-
-
Saturday 6th February 2021 02:14 GMT nerdbert
Re: At Jenny, re: config files.
Back in the days when Linux wasn't even a wet dream and when emacs meant "eight megs and constantly swapping" (i.e. when even workstations with 4 MB of RAM were considered top-of-the-line), emacs was the first program I installed on a workstation simply because it automatically made a backup of any file you edited with it. It saved my butt more times than I can count. Before that, the tape drive and OS tapes where my frequent companions when installing the latest workstation coming down the line.
-
-
-
-
Tuesday 9th February 2021 16:56 GMT Anonymous Coward
Software defined configuration systems
Since we're on the subject (and this is where it would be really good if the non-article sections of the forums on The Register were maybe a bit more prominent (and so, perhaps, would end up being better and more regularly used), or maybe if you could all let me know where you do all hang out for this kind of chat (StackExchange sites would sadly boot off such questions as "opinion based" or "asking for recommendations")?)…
Which software defined configuration system would any of you recommend for managing your boxen on a small home network (a couple of Linux boxen, and a Mac laptop and desktop (although probably not essential for the Macs to be under control))?
I'm "aware" of Puppet, Chef, Ansible, etc, but haven't tried any of them. We do use software defined configuration at work, but unfortunately it's an in-house and rather vintage system (to be fair, it does work quite well, once you get the hang of it, but the learning curve is painfully steep, and it requires a fair chunk of fiddly server setup to run the system itself). It would definitely be more useful for me to have something a lot simpler (and also a lot more widely used in the real world) for my home network…
-
-
-
Friday 5th February 2021 15:48 GMT GlenP
If the gate controller had been Windows then the regular reboots needed would have soon uncovered the bad config
When I came into my current role there was a PC sitting in a machine that hadn't been restarted for about 3 years. I left well alone! Eventually a few years later someone shut it down without thinking and the HD failed but I think it had gone around 6 years at that point.
-
-
Friday 5th February 2021 17:44 GMT Sykowasp
I think the first guy made the breaking change after being let go.
He knew one day the box would be rebooted, and then things would go wrong.
It only affected the gates, so not business halting, no money lost, so reduces the risk of a lawsuit should the change be detected as malicious (very hard to prove anyway).
Good work I say.
-
Tuesday 9th February 2021 16:35 GMT Anonymous Coward
«I'm thinking that the person who "managed" that box before did not deserve being laid off. He deserved being fired.»
Or it could be equally likely that manglement took the opinion: "Right, you've installed the shiny access control system, now on to the next thing… What do you mean, it'll need ongoing software maintenance from time to time…? I said, on to the next thing!!!"
-
Friday 5th February 2021 08:54 GMT GlenP
Had a call...
...from a previous employer some months after they'd made me redundant, "Do you know the admin passwords to the CAD machines?"
I was tempted to point out that even if I did know them the Company had decided my role was no longer required so obviously didn't need me to tell them the passwords. It was irrelevant, I'd never had those passwords as the systems were purchased, installed and managed through a third party not by IT.
-
Friday 5th February 2021 09:30 GMT Anonymous Coward
Re: Had a call...
I was made redundant from my previous job. It was an interesting experience, as the consultancy meetings about my role all went like this:-
HR/Management: "We think your projects are finished, so your role is no longer required."
Me: "They are nowhere near finished. The network speed dictates that the data moves for one client alone in Project C will require two months, even before we begin processing that data for ingestion at the new destination. There is months of work left in my projects, at least."
HR/Management: "We disagree."
Me: "You're disagreeing with your own network infrastructure. Good luck with that."
On my last day, my boss was out of the office (for very understandable personal reasons - not out of malice or ill will). He sent me an email: "Hey, we haven't started the main phase for Project C yet. Could you please provide us a quick high-level outline of how you'd handle each client during this phase?"
Naturally, I forwarded the email to my personal email address immediately. I then responded with a professional and courteous email.
I did consider taking this to tribunal, but I was lucky and got new job offers almost immediately. So I decided to simply move on with life. Asking for time off because you're suing your previous employer is not a good look at a new job! Also, it's good to just move on and not dwell on past unpleasantness.
But they were very lucky that I did get a new job so easily. Otherwise, they would have had a very awkward time explaining why my projects were still running, but my role was no longer there to run them...
(Posted anonymously, as I have no wish to identify the previous employer. The company's management may be crap, but the people I worked with were good and I have no wish to endanger them.)
-
Friday 5th February 2021 09:47 GMT GlenP
Re: Had a call...
I did consider taking this to tribunal
In the role I mentioned I had grounds for taking them to a tribunal but as they paid me a reasonable amount above statutory, untaxed money in lieu of notice, a few extras such as a "stationery allowance" as I wouldn't be in the office, etc. I didn't bother. I had a new job within 2 weeks anyway so the timing was right.
-
Friday 5th February 2021 13:54 GMT ICPurvis47
Re: Had a call...
"I did consider taking this to tribunal"
I was dismissed from my last employment on a trumped-up charge, but really because the bright young things didn't like employing a greybeard. I tried to reason with them, but they refused to co-operate and eventually suggested I take them to a tribunal. As it was Christmas, I let it wait until the new year, but then found that I had missed the deadline for the application, I had assumed it was six months, but it turned out to be only three. That probably explained why they took so long over the negotiations, they wanted me to run out of time.
-
Friday 5th February 2021 14:21 GMT big_D
Re: Had a call...
At one company, they made me redundant and then hired someone else straight away - against the law here.
Unfortunately, my lawyer was on holiday and his office said, it was no problem, he would deal with it when he got back. I had an appointment on his first day back... We had missed the deadline by 1 day! Grrr!
Getting a lawyer to sue another lawyer isn't fun. I had another gig, so let it rest and moved on.
-
Friday 5th February 2021 16:52 GMT Jonathan Richards 1
Almost correct information
> it turned out to be only three [months]
Assuming this to be in the UK, then the actual deadline is three months less one day
Ref: How early conciliation works [acas.org.uk]
-
-
-
Friday 5th February 2021 11:00 GMT Anonymous Coward
Re: Had a call...
Had something similar, only this time my job role (essentially deskside support, network, BI, DBA, anything with a plug) was got rid of and my position got downgraded.
New IT Manager comes in along with a contractor to take over the BI & SQL I used to do who then proceeds to ask me to help him understand the database structure, etc.
I refuse.
The IT Manager overhears this, goes balistic and drags me to HR. Fortuneatly they point out that it's up to me whether I want to help the overpaid contractor (being paid way more that I was before the downgrade) since the company isn't paying for that part of my knowledge anymore.
The best thing was it cost the company a ton in external support fees (~£5k IIRC) when I handed my notice in the following day since I'd had access to all of the systems and they wanted to make sure I hadn't left any backdoors.
-
Tuesday 9th February 2021 16:59 GMT Alan Brown
Re: Had a call...
"since I'd had access to all of the systems and they wanted to make sure I hadn't left any backdoors."
I've passed lists of all systems I had acess too and asked for written confirmation that every single one has has its passwords changed in order to ensure I have no knowledge of them or ensuling liability
-
-
-
Friday 5th February 2021 09:39 GMT Sam not the Viking
Re: Had a call...
Had a night-time call from my previous employer, actually their security/alarm company, six months after I had left. A break-in had been detected and the procedure was that someone needed to attend to investigate and be there when the police arrived (if a second trip was set). Explaining that I no longer worked there the security company explained that they knew that but none of the 'official' responders were answering their phone, 'Could I help?'
No. I sent an email to my old boss later in the day.
-
Friday 5th February 2021 10:07 GMT Pascal Monett
That was very stupid from the security company, and it could have landed them in court.
An ex-employee has no authority to be on company premises after hours - certainly not when an alarm had been tripped.
They could have unwittingly been setting you up - or wittingly.
In any case, refusal was the only appropriate answer.
-
Friday 5th February 2021 14:50 GMT Sam not the Viking
To be fair, I think they were hoping I had a number for someone who could go in. I only thought of that sometime later.
I had no keys or cards, but I would be 99.9999% certain the door-pass and reset codes etc. were the same.
I tend to attribute these things to the 'cock-up' school of management rather than a deep-thinking mastermind. After all, they are not BOFH trainees.
-
-
Friday 5th February 2021 18:28 GMT Anonymous Coward
Re: Had a call...
I had a similar story. Turned out that after I left, my old employer did update the security contact list. Shortly after that, the alarm company sold his business. The old alarm company was a one man shop who contacted our the monitoring and call center duties. When he sold, it was to a bigger company that did its own monitoring. When contact lists went to the new co, they got the seller's list (the original list with my info) instead of the call center's list (the updated one)
-
Sunday 7th February 2021 02:35 GMT Anonymous Coward
Re: Had a call...
A fellow retiree's previous job (now a long time ago) was as an engineer with the national telephone network. A new manager retired him early on the grounds of limited efficiency (he had a chronic illness that required occasional time off).
A few days later he was asked by the police to attend an exchange because of an "emergency". My friend told them he had just retired, "we know" they said, "please come anyway". It was a 10 minute car trip away and when he arrived there was a fire engine, a couple of police officers, the new manager, and a 'civilian'.
He was asked to go into the 'back room' of the exchange and 'check it', "Why can't the manager do it?" he asked "Because he doesn't have the required security clearance" he was told by the 'civilian'. He left the manager seething outside and returned a few minutes later and told them that 'whatever' was OK. Yes, he got paid in full for at half-day at his old overtime call out rate...
-
Friday 5th February 2021 12:46 GMT Anonymous Coward
Re: Had a call...
Oh good grief yes, the entitlement of some.
I had shrugged off more than one straw-that-broke-the-camel's-back moment, but finally decided that I'd rescue the tattered remnants of my integrity and resigned when a fit of firing mania overtook senior management, regardless of the consequences, something with which I refused to participate.
Some months later I got a call from the HR Director to say that one of those fired was suing the pants of them, and did I think he had much of a case? (I'll not go into details, but it also involved French employment law.)
I said "Yes, he has, and I warned you about it at the time. You told me to mind my own business"
[CLICK]
-
Friday 5th February 2021 14:16 GMT big_D
Re: Had a call...
Been there, done that. A couple of times.
Although I was made redundant from one job, only for them to hire a new admin immediately afterwards! 2 weeks later, the new admin wrote to me, to thank me for all the documentation I'd written on the systems...
They made him redundant a year or so later as well.
-
Friday 5th February 2021 14:26 GMT Pen-y-gors
Re: Had a call...
Now, personally I'd expect all my passwords/accounts to be cancelled even before I reached the door, so pointless being asked to sort a problem.
If they haven't then that is a big hole in their security
I remember in one job that we realised something was up (compulsory redundancies) one morning when we came in and saw the IT manager hunched over a terminal in the Security manager's office - deleting accounts! They had already asked for volunteers, and my hand had gone up (after I finished negotiating the package - I was also union rep). But that wasn't enough. The compulsory ones had letters delivered by taxi the previous evening, asking them to go to a hotel at a specific time the next morning, and not to go into the office first. There they met management and got their redundo cheque. They were escorted into the building after 5pm to clear their desks!
Personally I came out of it well. Had a phone call the night before the last day offering me a new job at 50% more dosh, start immediately. The smile on my face when I went in the next day to collect my cheque!
-
Friday 5th February 2021 16:20 GMT Electronics'R'Us
Re: Had a call...
I left a job once and after finding the new job atmosphere was depressing (virtually nobody ever smiled, as I recall) I left* and went back to the old job as a contractor (I was the only person who really understood the diagnostics for an enormous video on demand system).
The time involved was about 6 months IIRC.
I got my old desk and I had voice messages and my IT account still worked and had a few months worth of various crap in the email client.
Security - we've heard of it.
* Actually I got myself fired - this was New Jersey and when someone over the age of 45 (IIRC) was let go, state law at the time assumed the company had engaged in age discrimination so I got a decent settlement for agreeing not to pursue an age discrimination claim against them.
-
Saturday 6th February 2021 10:15 GMT ICPurvis47
Re: Had a call...
We had a round of redundancies, my whole department (Technical Publications) was made redundant and its workload assigned to the Sales Department instead. I was called into my manager's office, and he said "I have some bad news for you". I surprised him by saying "What, you mean I have to stay?" I had already secured three new positions with competitor companies, I just had to decide which to accept. Five years later, I was head-hunted back into my old job, with a considerable raise, because the Sales department hadn't kept the manuals up to date, and the company was facing several law suits for breach of contract.
-
Monday 8th February 2021 09:51 GMT H in The Hague
Re: Had a call...
"... the Sales department hadn't kept the manuals up to date, and the company was facing several law suits for breach of contract."
As I try to explain to my customers: producing good technical documentation is expensive, not producing it is even more expensive (due increased costs of customer support, legal hassle, lost sales, etc.)
-
-
-
-
-
-
Saturday 6th February 2021 04:31 GMT jake
Re: Had a call...
I implemented a four hour minimum for on-site visits in (roughly) 1990, a couple years after I went solo. Double on weekends/holidays. A few clients balked at the new rate ... I simply told 'em "Don't call me unless you actually need me".
A new issue arose. Convincing 'em to pay 4 hours for a one minute visit. The old TV repairman's maxim applied, "I'm not charging you for thumping your telly with a screwdriver. I'm charging you for knowing where and how hard to thump your telly, and for showing up to do it". The explanation seems to have worked ... although about three years ago a child CEO wondered why I'd need to thump a telly with a screwdriver.
-
-
Wednesday 10th February 2021 00:39 GMT jtaylor
Re: Had a call...
When I was independent, I preferred to give fixed-price bids for projects, but also gave an hourly rate for whatever they wanted done.
Some customers told me that my "competition" (often some service company with annual retainer and high turnover) charged rather less per hour. I replied that was grand, and if they wanted what the other company offered, they should hire them. I explained that I work too quickly to make money at the lower rate.
-
-
-
-
-
-
Saturday 6th February 2021 01:48 GMT Timbo
Re: Had a call...
I got made redundant at about 4:30pm on the last day of the month and with immediate effect (note: I hadn't done anything wrong - the senior management just decided to shut down my department).
The MD told me that I should "take a couple of days off and come back in on Friday" to collect my redundancy cheque. It seems that he thought that the day after I was made redundant, that the firm would still be paying me my salary and so it was in his good grace, to allow me take some time off.
I gave him short shrift and in no uncertain terms, told that I would NOT be taking "a couple of days off", as I'd be looking for another job !
PS: The firm was British and the MD was American - maybe the yanks had different laws back then (this was in 1995), but I know what the word "immediate" means and sadly the MD, one Mr R Lepper, didn't.
(I have no qualms about naming and shaming the twat...hopefully, Google spiders this website and this story might turn up in a search some day... ;-) )
(And posted as Anon, simply because he needs to be shamed and I don't !!)
-
-
Monday 8th February 2021 15:47 GMT Fruit and Nutcase
Re: Had a call...
Yes, there is a bug/feature gap where if you wanted to post something as AC, then you need to tick [Post anonymously] before you [Submit] - you cannot go back to the post and tick [Post anonymously] within the 10 minute grace period - it will not take effect.
One could argue that were a search engine spider reach the page just before you came back and made the post anonymous, then, there would be a record of your name outside of theregister.com
I reported this a few months back - when this happened to me, I ended up deleting the post and re-submitting with [Post anonymously] ticked right from the start
-
-
Saturday 6th February 2021 06:05 GMT jake
Re: Had a call...
The Yank was probably referring to the standard two week notice that is a normal courtesy over here. Some people work the final two weeks, some just take two week's pay and leave, depending on the nature of the job and the relationship between the company and the soon to be ex employee.
And yes, the two weeks notice takes effect immediately. It can be longer, and either party can give such notice. I once gave the Palo Alto Unified School District two month's notice, so I'd have adequate time to teach my replacement the basics. They made a VERY good reference on my cv ... The funny part of this one is that I was only employed by them for a total of three months prior (installed and setup an HP3000 Series 37 system over the summer break).
Chances are pretty good that you did yourself out of two weeks pay.
-
Monday 8th February 2021 10:44 GMT Down not across
Re: Had a call...
The funny part of this one is that I was only employed by them for a total of three months prior (installed and setup an HP3000 Series 37 system over the summer break).
I have a soft spot for HP3000. I have Series 39 with 2 x 7914 disks and 7970E tape drive. Everything connected via HP-IB of course. Whopping 2MB of memory. MPE V is fun and I quite like SPL although it helps to have the full manual set for information on all the intrinsics. It is what I learnt COBOL (well, the little bit that I learnt) on.
-
-
Saturday 6th February 2021 15:33 GMT Anonymous Coward
Re: Had a call...
Yes, US law is very different to UK law - and it varies by state as well. In some places it's quite legal to tell your employees as they're packing up to leave in Friday - "have a nice weekend, don't come back on Monday'.
At a previous job it was a source of friction between our MD and his colleagues at the parent company over in the US. Some redundancies needed to be made, our MD wanted to do the right thing and give people as much notice and help as possible, while his US boss wanted him to do the "don't come back on Monday" trick. That was only a contributory factor, but in the end our MD (and several other directors who were in the same family) decided their positions weren't tenable and they left - it was downhill after that.
-
-
-
Friday 5th February 2021 09:50 GMT Unoriginal Handle
Worked for an ISP many years ago. Was involved with auditing user accounts and configs on sensitive switches and routers which controlled a fair chunk of the ISPs and their customers connectivity.
On being told I was made redundant, I handed the SecurID fob in to a senior colleague the same day, and got an email confirmation from them to confirm. No way I was going to open myself up to all sorts of problems if there were problems.
My foreign manager, on the other hand, is a completely different story and couldn't understand why I wouldn't fly to Europe for a few days, in my consultation period, when I had job interviews lined up...
-
Friday 5th February 2021 09:59 GMT Evil Auditor
Or did you depart knowing you'd left an accidental timebomb ticking behind you?
Maybe a timebomb nut not accidental: I used to maintain a certain software. My departure had been scheduled and was known for about ten months prior. Several earlier attempts to hand-over the software remained unanswered and in my very last week suddenly they started to panic. Second last day I travelled to HQ and tried together with them to lift the DB. It failed, repeatedly. And my willingness to help ceased with approaching the departure of the train that brought me home in time for supper. Don't know what happened afterwards and neither do I care.
If you asked them I can imagine it would be something like: this a**hole left chaos behind. Good riddance!
-
Sunday 7th February 2021 01:44 GMT el_oscuro
I had something like that too. A new contractor bid for our contract which involved a large critical system. But they asked for all of us greybeards to take a 30% paycut. When most of us balked, the company made offers to us to work in another division. So several of us took the offer.
One of the systems that I managed had a service account which connected to servers from several different groups, and my admin account was the only one that had the correct A/D permissions. Several attempts to get the A/D group to set up a proper service account were unsuccessful.
So several weeks before the transition I informed the incoming management that they would have to replace my admin account. The very last thing I did was email them to *make sure you change this service account and verify the new one works before deleting mine*.
So 2 weeks later, I got a panicked call: "The system went down when we deleted your account." I was like "you deleted my account? Didn't I brief you both in person a follow up email? And why did you delete it instead of just disabling it?"
"What can we do?" "I have no idea. I have never managed A/D and don't even know who to call." Eventually they got it working, but over the next year I would get called in several times to fix their messes, all at my original salary.
-
-
Friday 5th February 2021 10:15 GMT Pascal Monett
I got a wierd call one evening
It was the very evening of my last day on a contract in a large Luxembourg company. The contract had been slightly difficult because, on my first day, six months before, I quickly assessed that the IT manager's right-hand man had something against me and was not shy of being clear on that point.
Well, I didn't need to work with him, so it didn't bother me much. I learned the hard way that you don't need to work with someone to be bothered by them - especially when they work above your glass cieling.
Suffice it to say that I was not sorry to see the end of that contract. Then I got the call, which went something like this :
Caller : "Hi, it's me. Um, we've had a problem with the database you were working on, would you happen to have a backup of the code ?" <cue antennas starting to quiver>
Me : " Hi. What happened ?"
Caller : "Well the server crashed and, when it came back up, that db was corrupted. We need your backup."
Me : "I'm sure I have a local copy on the desktop of my account."
Caller : "Yeah, but Security was quick to erase that before we could assess the problem."
Me : "Well, there's the backup of my account. Surely you can restore that."
Caller : "Well, no, there is an unplanned maintenance on the backup server - we can't use it. We really need your copy." <ok, this stinks now>
Me : "I'm terribly sorry, but I would remind you that the NDA I signed specifically forbids me from taking any code from your premises. I have no backup copy. You're going to have to wait for the backup server's unplanned maintenance to end."
Then I hung up. The caller had been a guy I worked with, who had generally been nice to me - or so I thought.
This had obviously been an attempt to frame me for doing something that I definitely should not (and didn't). Had I been stupid enough to take the code, and even more stupid enough to admit to it, I would have undoubtedly found myself in very hot legal water, and possibly the end of my budding career.
This event taught me two things early on : 1) never trust anyone when someone high up doesn't like you, and 2) always stick to the rules.
Useful lessons.
-
Friday 5th February 2021 10:22 GMT Anonymous Coward
Outsourcing
Ah yes, outsourcing. That oh so brilliant idea that a revolving door of "programmers/sysadmins" in another country can be just as effective as the people with 20 years experience with the system in question in the name of cost saving. If outsourced resource is just as capable as the home country workers but cheaper, wouldn't the same logic apply at C-level?
-
-
Wednesday 10th February 2021 01:00 GMT jtaylor
Re: Outsourcing
I worked at a company that outsourced their HR.
It sucked. Instead of asking for help* or advice from someone I knew, and who knew me, I had to call a number and talk with someone who could only look in my file and offer generic anodyne legally-safe guidance.
*I know, HR is not your friend, but when my employee healthcare plan made a mistake, HR person had a lot more weight then I did to get it fixed. Also, they do know The Right Way to do certain things in the workplace.
-
Wednesday 10th February 2021 15:41 GMT Anonymous Coward
Re: Outsourcing
My employer does this. To address an issue with benefits (like they miscalculated my HSA withholding again), I call a 3rd party call center that doesn't specialize in HR, who sends a message to a group mailbox watched by the real HR department, who sends messages back. Nobody - not me, not the call center - is allowed to talk to HR. It takes weeks or months to get anything fixed.
Of course, all the time I'm trying to get them to fix errors is paid time...
-
-
-
Friday 5th February 2021 10:41 GMT PerlyKing
Execu-barge
Not quite the same thing, but at a former workplace the company's car park was secured with one of those ramps that rotate up out of the ground. I got to work one day in time to see a large black BMW beached on top of it, as if someone in a hurry had tried to tailgate their way into the car park. Ouch.
-
-
Saturday 6th February 2021 17:44 GMT SImon Hobson
Re: Execu-barge
They have an especially savage set of bollards in Manchester - having tried various less dramatic ways of keeping people from driving down a particular road. All I can say is that some people are clearly not fit to be in charge of a vehicle - there is a LOT of very clear signage leading up to these bollards.
-
Friday 5th February 2021 11:24 GMT Mast1
Re: Execu-barge
Again, slightly OT, but talking of "beached BMWs", I saw one actually perform the beaching on an oil drum as he started to pull onto a roundabout. It had fallen out of the back doors of a (slightly battered) crew bus that had been in front of him and rolled under his bumper (fender). Rear wheel drive ensured that the BMW managed to rise up and go beyond the point of balance.
-
Friday 5th February 2021 11:36 GMT Doctor Syntax
Re: Execu-barge
A long time ago I had an MGB, an early one before they installed a suspension that made them look as if they were on stilts. Driving over a "sleeping policeman" (speed hump) outside North Queen St police station in Belfast it grounded amid ships. I had to get out to get more clearance to push it forwards.
I've never liked speed humps. I'm sure the accumulated wear & tear on suspension components must contribute to accidents but, as they're not happening at the actual places they were caused, they reasons go undetected. Rumble strips are a different matter; if yu take them fast enough the suspension filters them out.
-
Friday 5th February 2021 13:50 GMT Kubla Cant
Re: Execu-barge
I suspect most drivers detest speed humps. Question: if a road is safe to drive at 30mph, and a speed hump damages your car (or your neck) when you cross it at 30mph, does the liability for damage or injury lie with the party that built the speed hump? Sadly, I suspect not.
If my car is so low that a speed hump damages the underside, it's a case of "shit happens". Can I assume the same applies if I drive a bulldozer with the ram lowered?
Householders who are pleased when a hump is installed in their street should consider investing in a seismometer before they congratulate themselves. The effect of an HGV crossing a bump at speed is not unlike a small earth tremor.
-
Friday 5th February 2021 14:08 GMT ICPurvis47
Re: Execu-barge
We local residents had a long battle with the council after they installed speed bumps in a residential street that led to an industrial estate. The continuous procession of HGVs caused severe seismic vibrations that were damaging the houses along the street, not to mention the fact that said HGVs were being routed past a recreation ground, a creche, and an old people's home on their way into and out of the estate. I left some four years ago, but the battle still goes on, heaven knows what will happen when the first house falls down.
-
-
Saturday 6th February 2021 18:28 GMT Doctor Syntax
Re: Execu-barge
They're put in because they're required to enforce 20 mph areas. The possibility that they might cause damage leading to an accident outside the area isn't a concern because obviously the cause can't be traced back. The road safety industry is very good at denying its the consequences of its actions. It's currently working hard at denying that "smart" motorways* are dangerous.
At the same time they ignore dangerous features: I know of one stretch of road which has notices up along the lines of X casualties in 5 years (not very specific as to which 5 years as the notice has been there for at least 10). That stretch of road has 4 junctions with extremely poor sight lines including an oblique cross-roads with another A road which very clearly needs a roundabout. The only remedial action has been to reconfigure the major crossroads to make it even more dangerous and to provide onesmall, inconspicuous convex mirror.
*Probably somebody thought it smart to add an extra lane without having all the hassle and cost of widening the wayleave.
-
Tuesday 9th February 2021 17:43 GMT Alan Brown
Re: Execu-barge
"The only remedial action has been to reconfigure the major crossroads to make it even more dangerous and to provide onesmall, inconspicuous convex mirror."
90% of "safety fixes" on roads result in safety issues becoming WORSE
Traffic psychology is perverse - anythign which visibily "improves safety" increases driver speeds and have them paying less attention to their surroundings. If you want roads to be safer, you NEED to make drivers feel vulnerable
painted lines, fences, humps and crossings are particularly bad for this issue
-
-
-
Saturday 6th February 2021 08:48 GMT Giles C
Re: Execu-barge
I run two cars a standard Bmw and a tiger (lotus 7 style kit car). Speed bumps are a literal pain. I have been beached on a couple of them in the tiger.
But last night I had to go food shopping and there is a new speed bump on the way to the local Tesco. Well I say speed bump is it more like a row of bricks set on the road surface, I saw it in my headlights and slowed right down, you still felt the shudder as the wheels went over. The tiger well it would have bent the chassis rails.
As for those blocks of concrete they set in the road, it can’t get over them as the ground clearance is about 80mm..... and going over the middle would ground the engine as the drop is too high for the suspension.
-
Tuesday 9th February 2021 17:45 GMT Alan Brown
Re: Execu-barge
"Speed bumps are a literal pain. I have been beached on a couple of them in the tiger."
It's better to cause the road to be "waisted" by parked cars so there are areas where vehicles need to go single file AND visiblity is dodgy
Any clear road == "BOOTING IT!" - which is not what you want on residential roads
The laws are clear (even in the USA) - safety of residents of an area MUST NOT be compromised for the "convenience" of drivers passing through
https://nyc.streetsblog.org/2017/01/05/states-highest-court-holds-nyc-liable-for-injuries-on-streets-without-traffic-calming
-
-
Saturday 6th February 2021 20:26 GMT John Brown (no body)
Re: Execu-barge
"The effect of an HGV crossing a bump at speed is not unlike a small earth tremor."
On a particular residential street I use frequently, one of the speed bumps, the "pad" type you try to centre on to get the minimum lift as you cross it, the road has subsided a little on the trailing side. People who are not aware of this will attempt to straddle it as you do, then be rather surprised when the front end of the car dips further than expected and they bottom out or scrape the front bumper/spoiler.
As with most speed bumps in the UK, naturally they are built to the maximum height allowed for the roads speed limit so that dip makes them a few inches higher in practice than they should be and potentially in breath of the law until the fix the road.
-
Monday 8th February 2021 15:57 GMT Fruit and Nutcase
Re: Execu-barge
Householders who are pleased when a hump is installed in their street should consider investing in a seismometer before they congratulate themselves
And the drop in air quality with cars (at least with internal combustion engines for the immediate future) slow down and then accelerate away with increased emissions
-
Tuesday 9th February 2021 17:37 GMT Alan Brown
Re: Execu-barge
"Householders who are pleased when a hump is installed in their street should consider investing in a seismometer before they congratulate themselves. "
Speedhumps on such roads are usually because some bright spark decided traffic wasn't getting through and put in parking restrictions which result in the road becoming a rat run full of speedsters.
The REAL fix is to back out the alterations which caused the traffic issue in the first place, not add on more layers of shit - the story of the old lady who swallowed a fly could be about most council traffic policies
-
Tuesday 9th February 2021 17:49 GMT Alan Brown
Re: Execu-barge
It's worth noting this adage about traffic safety:
"Speed humps are almost always put in place to protect children walking to school from being killed by cars containing children being driven to school"
The real fix isn't to slow down the cars but to ask if they should be on THAT road at all
-
-
-
Friday 5th February 2021 14:25 GMT Martin Gregorie
Re: Execu-barge
The biggest and bestest speed bumps I've ever seen were on the outskirts of Abu Dhabi. These were at least 30cm high and 5m long, with roughly 1m long on- and off-ramps. They occupied the full width of the road bed, so dodging them was not an option. Even the local taxi drivers wouldn't tackle them at more than a slow walking speed, but many drivers obviously did, because there was a fine collection of interestingly twisted exhaust pipes and mufflers alongside each of them.
Unfortunately I was never there to see a driver donating a new specimen to the roadside collection.
-
Friday 5th February 2021 17:25 GMT Anonymous Coward
Re: Execu-barge
A few years ago I lived in a place which shall remain nameless, but it was quite full with expensive cars.
That got amusing when the council decided to put a speed plateau in (elevated section where roads cross) to slow people down, as it spent the first month of its existence filing down spoilers and undercarriages of sports cars..
-
Monday 8th February 2021 16:16 GMT Fruit and Nutcase
Re: Execu-barge
"The biggest and bestest speed bumps I've ever seen were on the outskirts of Abu Dhabi. These were at least 30cm high and 5m long, with roughly 1m long on- and off-ramps"
Kensington and Chelsea council should perhaps consider those for streets around Harrods, maybe ones which rise up from the road in the evening during the summer months when the owners of super-cars from the middle east ship them over for "the season" and use those streets as a drag strip and flout traffic laws and parking regulations with impunity to make life hell for other road users and residents alike
-
Monday 8th February 2021 17:33 GMT YetAnotherLocksmith
Re: Execu-barge
I think having them pneumatic or hydraulic would be a big goer. A radar detector or AI CCTV camera determines speed, and if over the limit, it abruptly raises the speed limiting device. Maybe make the lift proportional to the speed?
Very low cars would simply go slow and not need to worry as the road would be flat. Cars at 25mph, they'd get a small warning. But if they come through at 50+, they get launched into a special wall built just for the case, from the ramp that suddenly forms.
Buses and HGVs acting properly would also be fine. And, it would be really apparent that people were speeding, too, which would shame some into slowing down. Those that didn't slow down, well, they get it enforced upon them!
-
Tuesday 9th February 2021 17:51 GMT Alan Brown
Re: Execu-barge
"when the owners of super-cars from the middle east ship them over for "the season"
It's not il;legal to be in possession of a big bag of birdseed
nor is it illegal to scatter it over the roofs of such cars as they're parked on the roadside when you walk past
Grass seed for added fun as it works its way into crevices. Just make sure you don't use the kind which is treated to be poisonous to birds
-
-
-
-
Monday 8th February 2021 15:18 GMT Anonymous Coward
Re: Execu-barge
I was witness to a " near BMW beaching" being in the car at the time. At my old job we had a particularly obnoxious head of operations, so obnoxious that he bullied many people out of the business over 6 years including my boss of 41 years service. Anyway we were visiting an MSP to have a look at their DR facilities (waste of time but that's another story) my boss should have gone but was off sick so it was just me and the *twat* He always drove a new (on PCP) BMW, this was a 5 Series . The MSP were based in an old factory which has a large goods inwards with a ramp and shutter door so that lorries could back in. The Ramp led from the carpark of the building which was on a slope. So imagine the scene we're driving along slowly looking for a parking space we drove over the ramp (at 90 degrees) which was marked with yellow paint along its edge, but as it was on a slope one edge of the ramp was level with the carpark but the otherside wasn't!!! *CRUNCH* as we drove off the edge of the ramp, must have been a good 12-18" drop!!!! The Twat went mental, whilst I sat there trying not to piss my pants laughing. Hahahahah couldn't happen to a nicer bloke, wanker
-
-
Tuesday 9th February 2021 00:59 GMT Fruit and Nutcase
Re: Execu-barge
Over in Dublin...
'The Beast' felled by US embassy ramp
https://www.independent.ie/irish-news/the-beast-felled-by-us-embassy-ramp-26735700.html
The BBC reported that it was a "spare limo"
https://www.bbc.co.uk/news/world-us-canada-13507338
icon: Marine One
-
-
Friday 5th February 2021 10:50 GMT ColinPa
The weakest link
A customer told me of an incident where he worked. They were a top z/OS shop, with two hot sites etc, they tested failover every 2 weeks etc. One of the critical applications told people what fresh fruit etc to put in the containers. It was very slick. The little problem they had was when the drivers came to pick up the containers and collect the delivery instructions, and the printer went down. This printer had been in the cubicle for 10 years. The driver would arrive, the person in the booth would print the instructions and off goes the driver.
When the printer went down there were lorries backing up along the road. As they could not buy a replacement printer with coax attach, it took over a day to get it working, as it was out of wifi range etc.
For want of a nail....
-
Friday 5th February 2021 11:51 GMT PickledAardvark
The final days...
When you are leaving, every problem becomes somebody else's problem, with you sitting over their shoulder to advise. For a couple of days at least, you shouldn't be doing anything, just sitting around answering questions and showing off tricks. That is the sensible world...
But, aah, the glorious final hour. Hand over the list of test and demo accounts you can remember using. Log in for the last time as admin, and remove all of your privileged accounts from privileged groups. And pray that you never set up a service account using any of your accounts.
-
Friday 5th February 2021 12:03 GMT Anonymous Coward
I don't support the barrier sorry
I had to do a site visit and got roped into a "can you look at my computer quickly?" for the receptionist on the front desk, after a few minutes she walked off and was nowhere to be seen for quite some time while I worked on her PC. A few minutes after that a buzzer sounded from somewhere further down the desk, then again a minute later, and again, getting substantially more aggressive with each press. This was apparently someone trying to buzz their way out of the car park. About five minutes later a red faced man came stomping in, saw me behind the reception desk and assuming I was the receptionist launched straight into shouting directly into my face...
The receptionist did come back a minute into his rant, promptly threw me under the bus for not opening the parking barrier, and told me she assumed I knew how to do it as I worked in IT. I didn't even know the buzzing was someone trying to exit the car park, or that I was supposed to be covering her job!
People seem to get terribly upset about parking barriers!
-
Friday 5th February 2021 12:23 GMT A.P. Veening
Re: I don't support the barrier sorry
The receptionist did come back a minute into his rant, promptly threw me under the bus for not opening the parking barrier, and told me she assumed I knew how to do it as I worked in IT.
At that point I would probably have lost my cool and turned it into a complete shouting match, most likely leaving that receptionist in tears for being a bloody incompetent, can't even take care of a PC.
-
-
Friday 5th February 2021 12:03 GMT TonyJ
Device Isolation
I never mix my work and personal machines/data other than I will have a VM for work on my personal machine.
I don't use my personal Onedrive/Seafile accounts on those machines and when the role is over, it's archived to backup and removed from my laptop. If it's particularly sensitive then it's wiped outright.
And because I use Hyper-V as it is good enough for my needs, I get the added bonus of no camera passthrough so when I am asked to "turn your camera on please" in MS Teams meetings I can politely decline as the machines doesn't have one and no lies are being told.
Likewise if I have to use a customer-supplied laptop I will take an image of it on delivery and re-deploy said image prior to returning it, with an archive of files placed on their for their perusal should they so wish.
I've found this works well even from the perspective that there are no random pictures/emails etc from friends on there. Complete isolation.
-
Friday 5th February 2021 17:18 GMT Anonymous Coward
Farewell email to all - even though that was blocked..
When you know in advance you're leaving because you're engineered out of the company, you have time to plan.
The company in question had an incident a couple of years earlier when a very naïve member of staff objected in a company wide email to a tax saving plan that was, umm, let's stick with "creative". Needless to say, that email didn't go down so well with management, said person was marched out of the building and that email vanished from everyone's inbox (except mine because I had dumped it in text for the hell of it, but let's stay on track). Anyway, that was the end of the email to "all" - its use was blocked.
Except, if you divide an organisation in departments and then maintain mailing lists to every department member it only requires emailing to the collection of those lists to achieve the same result. Even more entertaining is to flip the "read" flag, which - amongst others - told me that the chairman read my departing email three times, which meant he must have seen it at least once (as his secretary would have triggered the first one).
The email itself? As I said, I had time. It was deemed amusing and in general appreciated. Despite the engineering which was pure HR politics, I worked with some great people so it was meant for them.
As for the aforementioned naïve chap, he got his own back.
He genuinely became a millionaire :).
-
Friday 5th February 2021 18:24 GMT Old Used Programmer
The Joys of Departure
At one company that did a large "reduction in force" (i.e. round of layoffs), I was told that the person told to pick up everything I was doing was given condolences by people who knew both of us. I had accumulated a large number of miscellaneous recurring--some periodic, some aperiodic--tasks. I did my honest best to turn over everything to my replacement. However, I heard back from colleagues still there that for about the next year, every time some odd thing turned up that needed attention and the question "Who takes care of that?" came up, my name got prominent mention. It wasn't any *deliberate* time bombs, it's just that I couldn't, off hand, remember everything I'd dealt with, so if any of those surprises had occurred when I was there, I'd've just dealt with them before anyone else noticed.
-
Friday 5th February 2021 21:49 GMT Anonymous Coward
I'm still (emotionally) supporting the guys I left behind
They closed down all but one of the sites I supported. As the head of the team with 20+ years experience they wanted to keep me on to support the remaining site.
turned out I had forgotten to sign all the contract updates, so my notice period was only 1 month ... and all the servers were stored at one of the sites set to be demolished.
I threatened to quit if they didn't make me redundant, and they thought about it for about 2 seconds before giving in.
I had to employ an entire team to replace me, but they look after all the stuff I left behind. good bye to the 24/7 lifestyle where users called my personal number whenever they felt like it ... however I got on so well with the new team, that I still talk to them now ... it's only been 1 year since I left, and I'm still giving them hints over how to fix the kit.
I think I might need to start being mean/changing my email/mobile number ...
-
Friday 5th February 2021 22:35 GMT ShortLegs
Ah, the joys of leaving
Many, many (ok, just over two) decades ago I worked for a major (and I mean major, contributed 2.2% gross gdp to UK economy) UK company, via a outsourcer. This client was the "jewel in the crown" for the outsourcer, and as such the SDM and I were often the subject of attention from senior management. Times were great, the client was awesome to work for, and about a year after starting received a second, and very large, promotion. Things were going swimmingly...
...until the new service delivery manager arrived, complete with a team of four, replacing the old SDM. One of the new arrivals, let us call him Chas, had the same role as me... ooops. So I could have bought a claim for dismissal, but - and bearing in mind that in 1997-99 jobs offers arrived at the rate of 3 per day on a slow day, I made plans to move on. imed to perfection: the weekend I left, client was migrating the entire UK infrastructure from flat TRT to a routed IP network, switched ethernet, etc. The young gent taking over from me was crapping himself. Between him, I, and my oppo on the client side we arranged I would come in on the Monday and Tuesday as a consultant, with a suitable figure daily fee.
Late Tuesday I hand Chas my invoice. His face twitches, then says "ah, but I dont need to pay this. You see, I never processed your resignation letter".
"Tough, I start a new job next Monday" I should add that at this point I still had over 30 days outstanding leave...
Fast forward to the end of the month. Pay slip from the outsourcer. No pay-in-lieu. Invoice unpaid. Call Chas' boss, whom I had a good working relationship with. Who was horrified to learn I had resigned, tries to persuade me back, and on learning 'Chas' was my manager, agrees the screw-up is nothing more sinister than incompetence. Asks me to send invoice.
End result? That months salary kept. Invoice paid following week. Bonus? A further months pay as the resignation didn't 'officially' occur until David received my notice, and thus a further months pay (and pay for two further day's in lieu).
Icing on the cake? Client had 'Chas' removed from the account. I didn't laugh much at all :)
-
Saturday 6th February 2021 18:37 GMT Doctor Syntax
Re: Ah, the joys of leaving
"ah, but I dont need to pay this. You see, I never processed your resignation letter"
What he did or didn't do with it would have been his problem. Your resignation had been presented to an appropriate representative of your employer (assuming he was that appropriate representative). Anything further would be between him and the rest of the company.
-
-
Saturday 6th February 2021 00:52 GMT MachDiamond
Told off for it
When I gave my notice to an aerospace company, the unqualified CEO (it was a small firm) didn't open the email I sent with my "take this job....." notification until a few days before I was leaving. One of the issues I had with the company was the grief I was given for "taking too much time with documentation". As the head of the department of one, it took time for the company to get around to finding a replacement. Software is good, but you have to have somebody designing hardware for it to run on (embedded stuff). A little birdy told me a few months later that the new guy, that was universally despised from day one, was gushing over having such great documentation that let him get up to speed quickly on everything except the oldest machine which I was specifically told to not do any work on because it was being retired. Of course they didn't retire it and wound up having to pay all sorts of overtime to give it a total makeover and were hampered by the fact that all of the documentation that did exist didn't match up with how it sat. I put so much work on it that I could have sat down and done most of the wiring diagrams from memory. It did need a complete rebuild as it was the company's first successful machine and had been revised so many times there wasn't a wire longer than 600cm on it's 3m height. I remember having to pull out a handful of crispy wiring that had been inside the control box for at least a year after one mishap. The person before me just wired a repair around it and left the bad wiring in.
Many times, your dedication and attention to detail isn't going to be appreciated by corporate until long after you are gone. At least you get some satisfaction that it wasn't your fault when the car park gate tries to slice the execu-barge in twain.
-
Saturday 6th February 2021 11:53 GMT Anonymous Coward
Got made redundant a few years ago
Headed up a small team for the companies main product. Been there 20 odd years.
Project manager knew next to nothing about software engineer or the product.
The engineering manager had a hardware production background, also knew very little about software engineering. Said I was too expensive and my skill set could be replaced by anyone and that I was unable to learn anything new...
Everyone else they made redundant was gone in a week. I was still there on my last day 3 months later as they were attempting to get me to fix all the long standing bugs before I went...
Walked straight into a job paying 10k more with less responsibilities.
Needless to say 4yrs later they are still trying to find people to replace me and they have failed to release a new version of the product.
-
Tuesday 9th February 2021 18:04 GMT Alan Brown
Re: Got made redundant a few years ago
"Needless to say 4yrs later they are still trying to find people to replace me and they have failed to release a new version of the product."
You see this a lot on the customer side.
Needless to say if you find a supplier has just had a large change of staff, it's time to start looking at alternative products or risk being left high and dry
-
-
Saturday 6th February 2021 12:17 GMT Will Godfrey
Sometimes you can be made redundant for your own benefit.
This happened to me something like 20 years ago. I was informed I was being made redundant, along with a couple of other engineers. No official explanation was given, but we were privately told by our section head to keep our heads down, and it would be to our benefit.
Well the day came, and we got full pay, allowance in lieu of holiday, and a bonus. A month later the firm went bust. It seems the boss (knowing this was going to happen) wanted to help those that did the real work.
-
Saturday 6th February 2021 17:50 GMT Anonymous Coward
Re: Sometimes you can be made redundant for your own benefit.
Im hindsight being made redundant was the best thing my last place did for me. By the end it was so toxic that the three of us being got rid of had some fairly serious health problems. At least by making us redundant they had to pay redundancy - but not a single penny more than the law dictated.
-
-
Saturday 6th February 2021 15:28 GMT Anonymous Coward
Anon for obvious reasons ...
When I left my last place, I left everything running smoothly - disks big enough to hold logs, and rotation set up to purge old ones, that sort of thing. And after much scavenging among kit discarded by the guys running Windows stuff, I'd finally got reasonably reliable hardware and decent disk space. I even had enough room that I could have shifted VMs off any of my hosts in the event of failure - enough on the other 3 to take up the slack.
About a week after I left, I heard from a colleague that one of the hosts had failed. Shouldn't have been a problem - just temporarily stuff the disks in something else and transfer the VMs to another host, longest task would have been copying the customer mail stores which takes a few hours. Anyone with a modicum of Linux admin experience could have had everything back up the same day.
However, the "manager" that made the last of us redundant was a) a complete and utter orifice, b) thought he was gods gift to system admin, and (you've probably already guessed this) actually knew SFA about systems admin and even less about Linux admin. I gather it took him a week to recover, during which time hundreds of customers were without mail - and I can only assume he told them a complete pack of lies about the cause of the failure and the cause of the delay. The one thing that was guaranteed with him is that I didn't need to worry about negotiating a fee to go in and fix it - as a matter of principle he'd have never had considered admitting that I did actually know stuff he didn't by asking for help.
He later killed the DNS for over 100 domains by ripping out the last of anything I'd put together. About a week later (give or take), the off site secondary servers he thought would "just carry on" expired the zones. I'd have been having a right good laugh at both these IF it had just been the company affected - but when it's affecting clients, many of whom I knew personally, then that's not a laughing matter. I was half tempted to ask some of them what lies he was telling them, and tell them the truth - but I decided against it.
-
Tuesday 9th February 2021 18:06 GMT Alan Brown
"I was half tempted to ask some of them what lies he was telling them, and tell them the truth - but I decided against it."
All you need to do is contact the the clients and tell them that critical staff aren't there anymore. Don't offer to assist or direct them to competitors. Just the heads-up is usually enough to start them worrying
-
-
Saturday 6th February 2021 17:56 GMT Greg 38
Similar tale
At a former company I'd worked, maintenance turned off the wrong machine when attempting to perform some routine work. The unfortunate machine was a 20yr old Applied Materials etcher used in chip making. When they realized the mistake and turned it back on, it wouldn't boot. The company's contract with Applied had been allowed to lapse and along with that all the detailed knowledge of coaxing the beast through troubling computer issues. A contractor had to be brought in and ended up replacing the floppy drive with a SD card reader and card formatted to look like a 1.44Mb floppy disk imaged with the boot files. After 2 weeks of reconfiguring the machine, it was back up again and the company out many $100k.
-
Sunday 7th February 2021 21:59 GMT Anonymous Coward
Re: Similar tale
I well recognise that issue, working in a chip (and more) research dept with much ancient kit, many back-ups where they exist are on 5" floppies and operating systems come on a stack of them. We lost one machine to a brown-out not quite deep enough for the building UPS to take over, when we finally got the PSU working and we then discovered the hard drive wouldn't spin up again, with magic smoke from the drive board. After much searching by the technicians of their home stashes of old kit we had used drive that might work, being an ex-IBM H/D build line tech I had to strip and manually check the disk pack's motor bearings weren't stuck lest we blew the 'new' (used) drives board trying to start it, reassembled and fitted it spun up and over several days was loaded with O/S and required software, only to discover the back-ups were the previous version and following an undocumented hardware update wouldn't work. The responsible academic who to save costs had cancelled the service agreement when the hardware update had been done had to go cap in hand to the machines manufacturer for the correct 'updated' software, that cost him dearly, both in loss of face and many thousands of pounds in charges that would have less if he'd paid for continuing service... A too familiar story in 'academia' and one that's getting worse.
-
Monday 8th February 2021 15:48 GMT Anonymous Coward
Re: Similar tale
seen this kind of thing so many times after working in a lab for 20 years. The boffins would buy some kit and it would come with support for 1 maybe 3 years. Then 10 years later when the old computer connected to said kit died they be stuck scabbing around for something just as old to run their old copy of the software as they hadn't had any updates for the last zillion years as they didn't bother paying for the maintenance contract.
-
-
-
Monday 8th February 2021 16:02 GMT Fr. Ted Crilly
];-)
should a 'fixed' it and got it back up and then redo the old changes and leave it as is awaiting the next unplug hero. (or better still rearrange to wall plugs back to the same inconvient arrangement as before, who me? not me I got it stood up again look its ancient and decrepit, I warned you you needed to spend money on it when i fixed it....)
-
Monday 15th February 2021 17:09 GMT Anonymous Coward
Out with Head High
Sounds familiar. I was the Quality Data Manager for a Fortune 100 company in 2008. We produced lots of widgets at the rate of about 8000 per minute per machine, so there was lots of data to keep up with in QA. We had no formal IT support, so everything I did was patched together with duct tape and chewing gum. The company decided to outsource/downsize our department from 150+ to 13 people, and I was not one of the 13. Three years earlier, the company had asked me to get a master's degree in Statistics, so after three years of full-time school, work, and a young family, to say I was burned out was an understatement.
At this point, I'd been with the company for 27 years, and you can imagine the environment in September 2008 - right before the 2008 US election and the economy was going to h*** with the financial crisis, etc. For whatever reason, it wasn't an immediate walk out the door. I worked for three weeks - until the end of the month - before leaving. I heard of other people that had, for example, the password to a mission-critical Access database (poor decision!) that never came back and refused to tell them the password. Not sure what they did there.
I decided that I'd spent 27 years building a reputation and wasn't going to leave on bad terms. I made sure everyone knew everything I was doing, updated status on everything I was working on, and concentrated on finishing my degree (Did I mention that I was in my final semester?) I backed up everything as IT was notorious for letting expensive data disappear without a trace after some threshold.
I suppose it really didn't matter - instead of termination, they kept us on an "inactive payroll" until our severance ran out. Terms were sweet - two weeks of severance for every year of service, and you accrued vacation and increases during that time. And they paid out vacation at the end of it all. Financially, it sucked because nobody where we live pays like they did. In 2021, I'm still not making what I did there in 2008.
It took two years to find employment again, but was only unemployed for about 5 months of that. The company has been since acquired by a large UK company and most of the manufacturing has been moved to offshore plants.
Whatever happened, I could at least sleep at night.