Service desk tech saved consultancy Capita from VPN meltdown, got a smack for it Welcome, dear reader, to yet another instalment of On-Call, The Register's weekly column featuring readers' tales of being asked to show up and save the day. This week, something a little different. Readers usually ask us not to identify the scene of their adventures – but the On-Call inbox recently received a tale set at …
Maybe it was his use of that bizarre and unloved compression format, RAR, that was the problem. I guess it was just an easy way of creating a self-extractor back then (and the less said about a random self-extractor downloaded from a newly registered domain having unfettered access to root drive the better). Can we all agree that there no sane reason for anyone to use RAR today, though? Right? Can't we?
[runs away expecting torrent of abuse for some reason]
Not sure if this is still the case (?), but they used to have an hilarious versioning system where they'd add a 1 on to anything that they patched, resulting in version numbers such as 6.8984.11111111111, and patch notes talking about version 6.8984.11111111111 being replaced by version 6.8984.111111111111 (and so on).
Not so much, but smacked down for pointing out why there was a problem, and having meeting notes that show I said it would be a problem before implementation.
Nobody in management likes a someone smarter than them, especially when they are below them on the food chain...
A bad manager will ignore your concerns, then smack you down for this when you're proven right.
A good manager will see this as an opportunity, not a threat and will both publicise when a concern is raised, and will promote the fix/brag about how their team helped solve the problem.
As a former tech for many years turned manager, I like smart people in my team. Helps to avoid shit hitting the fan in the first place, and helps to recover from a mess if things do go wrong. Only bad managers dislike those smarter than themselves further down the chain.
Several jobs ago at a small privately owned Beltway Bandit:
Owner has a staff meeting on Monday, directs us to explore option X by Wednesday to solve an urgent issue. We explore it, reject it as untenable. At the Wednesday staff meeting, Owner foully curses at me for exploring option X as he directed. My gutless supervisors (who were involved throughout) sit there schtum. After the meeting, they criticize ME for objecting that the boss had ordered the exact course of action he was disparaging. I privately dubbed them "Frick and Frack, the Spineless Brothers."
The company owner was one of those people who is never wrong even when he is.
I left for saner pastures very soon thereafter. The boss asked if there was anything he could do to persuade me to stay. I think I deserve a Congressional Medal for not responding, "Seppuku would be a good place to start."
- Anon because some of my erstwhile colleagues probably read The Register.
I had this situation when working for a UK Government Department (during the Bliar Era). Our company were asked to evaluate some software that had been procured for use by the NHS. My team diligently tested, analysed, tested again, and rejected this Crapware that had been bought from one of the Government's favoured suppliers, and the rejection was reported (by me) at at a high level meeting in Whitehall shortly afterwards....
Of course it was Windoze-based, and - of course - it had no intrinsic security whatsoever. It was to be used to record, store and retrieve confidential Patient Records across the entire NHS estate. Of course it didn't work properly, and of course it made it trivial for any "malicious actor" to recover, modify or even delete any data that had been recorded in its poorly specified and even more poorly written SQL Database. Moreover, it would require extensive (and expensive) training for the grunt-level administrative staff that were to be tasked with loading all the data on to it.....
The vendors of this rubbish were rubbing their hands with glee when the then Minister for Health gave the go-ahead for the roll-out of this disastrous mess.... It was at this point that it was discovered that the desktop computers used throughout the NHS were insufficiently powerful to be able to use this junkware, so a "procurement process" began for much higher specification machines. The high prices were "justified" simply by the need for their addition to the NHS Hardware Asset Register - requiring individual tagging (they would never have thought of using a Serial Number already on the machines!.....
Needless to say, I was hauled up before a Government Minister and the high-ups in our company, and given a dressing-down for my "Antagonistic and Negative Attitude" and my refusal to go along with the obviously very lucrative (for some!) plan that had been hatched. My Annual Bonus (>75% of my pay) was cancelled, and the amended terms of my contract (that had been modified without reference to me) were pointed out. I was not allowed to disclose any of this malfeasance, and not allowed to work for any company in a similar line of business for five years after leaving this shower.....
The roll-out of this disaster began shortly afterward, and the results we had predicted came to pass shortly afterwards. Many people involved in the fiasco made £MILLIONS and those of us who warned against it changed employment or left that line of work altogether soon afterward.
£BILLIONS of public money was squandered.
Anon - because I still have records of where the money went, and because I don't want to "Go for a walk in the woods...."
This post has been deleted by its author
I know of someone who had this happen to them too... a certain security vendor had a glitch happen to them around 10 years ago that marked *any* software with the word 'update' in their name as dodgy (which included their update engine), and where customers had opted to delete or quarantine those 'dodgy' executables, completely screwed up their machines. A signature update rolled back those problems, but where the update engine was screwed, those signatures couldn't be put on the machines. One of their engineers figured out how to fix the update engine (standard Windows installer command-line apparently), and together with their helpdesk widely saved their customers.
Apparently the engineer then got the boot a year or so later when they decided to lay off a significant chunk of their engineers with the VP of Engineering (a Yank) making it clear they wanted that engineer gone.
Anon, because obvious...
You have to wonder what happened to the people who implemented the change without testing on the platforms that were in use.
Sounds like the testers all used windows 7 and ‘forgot’ about the XP users.
At the very least as part of the post imp testing it should have been tested on both platforms so it could have been rolled back inside the change window
In my experience, half of them are exactly as you describe and the other half will complain if a desktop icon moves slightly. If you say an update is needed, they'll do lots of things to prevent you from installing it and making them learn anything different, no matter how many warnings you give them (correct or not) about things breaking catastrophically if they stay with the old version.
This is independent of the platform they use. The people who say they must use Windows [specific version] and no other version of Windows (mention Linux and they will not understand you, but once you explain yourself they'll have none of it), and the people who would never consider that anything not made by Apple could ever function may be different people, but they act the same. I imagine they're the same kind of people who refused to adopt computers when they were new, but fortunately I'm too young to have seen most of that.
I'm a PM with experience of managing tech and development teams. I have the joy of dealing with very senior managers and am often taken along as the subject matter expert. I've been slapped down many times in my career normally by senior managers who are busy promising things that are impossible to deliver or committing to timescales which are unachievable whilst refusing to pay overtime. My most memorable one was being told by a senior manager 'you made me look a right dick in there' and threatened me with the sack if it ever happened again. I kept my mouth shut but was thinking actually you did that for yourself. Needless top say I was already looking for another role.
The second word is fine(ish), but it is the first word that people have trouble with, and why it's often lengthened by adding an extra 'r'
The government and other public bodies pay them billions a year to run public services, along with giving them new contracts and contract extensions, despite the government's own Public Accounts Committee calling them out for failures, missed targets and poor performance
Private Eye gave them the nickname 'Crapita' many years ago and has a news item about them in almost every issue
This post has been deleted by its author
This is a long story, so I shall try to truncate it.
Basically, I was asked by my employer on Thursday if I could get to the States on Monday. After a bit of toing and froing, I agreed and applied for the US Visa I was told to apply for. The trip was uneventful - literally as the person I was meant to meet there didn't turn up, so I spent the few days there doing what I could. I had flown out on the Sunday and was coming back late on Thursday due to family commitments back in the UK. This had been agreed beforehand with the proviso that I would go out again, just let me know when you need me etc.
So, not only was I accused of "walking off site without permission" by management, but after a couple of weeks back in the UK, I was threatened by HR of gross misconduct as I had applied for the wrong Visa. All my explanations of timescales and protestations fell on deaf ears and eventually (that is after about 30 minutes of him arguing my case on my behalf) my team leader finally managed to get HR off my back.
On the plus side, I managed to get my wife a lovely Christmas present in the airport waiting for my flight home. Needless to say, I never volunteered for anything like that ever again and I am most fortunate now to have left that outsourcer. (No, not Crapita, but Shitpro.)
It's absolutely clear what happened here. Somebody further up the chain made promises on your behalf that did not align with what you'd been told (the usual bollocks of over promising no doubt). That somebody got a royal bollocking and rather than taking it on the chin and admitting their error they decided to blame you. Their intention may not have been to drop you in the shit with HR, but that is often an unintended consequence of such action.
Middle management coward is getting shit from senior management, middle management coward blames junior member of staff for the cock up, senior management take this straight to HR without any further discussion. Middle management coward is contacted by HR about what is now a disciplinary matter and rather than putting their hand up to the cock up they take the cowards way out and double down on the accusations.
Which is why you keep records of everything you've been told to do, and what your response was. Sometimes, I've included bcc'ing emails to my personal account because I knew the [expletive] in charge did actually have enough tech nouse to be able to selectively remove emails from the company system.
UK perspective: I didn't bcc to personal account as that would be breach of confidence/security issue (education sector, dealing with students under 19 &c) but I did print out key emails and initial them with date and keep them in a file. Only had to produce printouts a couple of times in a decade and a half.
This post has been deleted by its author
"Sometimes, I've included bcc'ing emails to my personal account"
Which is dangerous. If they have the desire to go into the system and selectively delete your email, they have enough technical knowledge to notice that BCCing, which is almost certainly an explicit breech of contract and it doesn't have to be for it to become a sackable offense. I get why you want to do it, but the last thing you need is to shoot yourself in the foot while keeping the evidence. You could save a copy onto your work computer where they wouldn't look for it, or if necessary keep notes or copies on paper, but sending it onto a non-approved system makes it much easier for them to penalize you whether or not they could for the original problem.
When I went over to Aus on business the travel company sent me a link to apply for the relevant visa, which they would pay and then recharge. When I looked I realised immediately I didn't need the one they were suggesting, the free e-visa was correct for a Brit for the purpose of the visit.
Because I'd clicked on their link, however, they charged us somewhere around £65 for the visa fee and "handling", neither of which were applicable. I told the person responsible but I suspect it was paid anyway,
This is a long story, so I shall try to truncate it.
Basically, I was asked by my employer on Thursday if I could get to the States on Monday. After a bit of toing and froing, I agreed and applied for the US Visa I was told to apply for. The trip was uneventful - literally as the person I was meant to meet there didn't turn up, so I spent the few days there doing what I could. I had flown out on the Sunday and was coming back late on Thursday due to family commitments back in the UK. This had been agreed beforehand with the proviso that I would go out again, just let me know when you need me etc.
So, not only was I accused of "walking off site without permission" by management, but after a couple of weeks back in the UK, I was threatened by HR of gross misconduct as I had applied for the wrong Visa. All my explanations of timescales and protestations fell on deaf ears and eventually (that is after about 30 minutes of him arguing my case on my behalf) my team leader finally managed to get HR off my back.
On the plus side, I managed to get my wife a lovely Christmas present in the airport waiting for my flight home. Needless to say, I never volunteered for anything like that ever again and I am most fortunate now to have left that outsourcer. (No, not Crapita, but Shitpro.)
I can sympathise with that, we used software products from a company that we were Guinea pigs for. Once had a nice chap of German descent at our offices trying to install a new version of their software for me to Beta test. He’d arrived that morning, they’d booked him in to stay with us for the day supposedly flying back that afternoon. He’d flown overnight from the USA with his laptop, a few CD Roms and assorted cables. He also brought with him a bad case of Jetlag and an attitude to match when things started to go wrong.
Things went wrong rather quickly because the freshly burned CD Roms were not allowing him to install his software in my test PC. He blamed my computer and then to demonstrate got his laptop out and tried it on that. That didn’t work either which he was seriously unhappy about and lead to more grumbling. He said he’d have to contact the office in the USA to see what was going wrong but no one would be there till after lunch in the UK.
I suggested having lunch and then worrying about contacting those across the pond afterwards. He reluctantly agreed and asked about where he could buy it. I mentioned there were a few sandwich bars nearby, a few restaurants and we also had a staff canteen where I was going. He didn’t seem keen on leaving the building but had questions about the canteen.
“Do they sell Sandwiches?” - Yes
“Do they have non carbonated beverages?” - Yes
“Can lunch be bought for under £5” - Yes
“Do they supply receipts?” - Yes I believe so
“Are there places to sit down?” - Yes
Those answered we ventured to the staff eatery for lunch. He explained that his company has a $10 meal allowance for lunch when “on the road”. He needed the receipt to be able to reclaim this from the firm. Furthermore because nothing was installing and he hadn’t identified the issue, he was likely to be here overnight. This was an issue because he was booked on a flight back later that afternoon to the USA. He had to get authorisation to change that and find + book a cheap hotel room for the night. Buy some underwear and a toothbrush as he hadn’t brought any clothes/wash kit with him. On top of all of that he had to explain to his wife that he wouldn’t be back that night - hardest one to do.
He talked to head office after lunch and eventually (I got the feeling blood from a stone would’ve been easier) got his authorisation. His wife was really quite unhappy given the shouting coming out of his cellphone. Sadly staff at headquarters didn’t have a clue as to why it wasn’t working. I said I had to go to a meeting and left him with the IT department phone number and another member of staff as a chaperone. When I got back he had installed everything and was just customising the system for us. I asked what had been wrong and he muttered something about truncated file names. It turned out that one file name was missing a digit at the end which had been truncated on the CD Roms. This stopped the installer as it checked all the files early on in the process and stopped when it couldn’t find one.
On the plus side as this was now a two day trip he had a further full $20 in expenses, to spend on dinner. He said that he needed to buy his wife a nice souvenir to placate, her so was off to Harrods. The new software was described beautifully by a colleague of mine as “Not even a Beta version at the moment, who wrote this buggy piece of crap”.
We had a tool which "helped" us decide which visa we needed for different countries. It wasn't very good. It said a UK citizen needed a visa to go from the UK to France, and could get it from the French embassy in Washington!
It also said I needed a work permit in China if I was doing any "education". This work permit takes 3 weeks + to get, and you need an address in China. As I was flying in Saturday morning - doing a "on support" Saturday night and flying home Sunday - I didnt have time to get an apartment or apply to the local police station which is only open Monday to Friday!
After weeks of complaining about the tool - they changed the tool to say "UK citizens do not need a visa to go to France"!
Every one else ignored the tool!
It said a UK citizen needed a visa to go from the UK to France, and could get it from the French embassy in Washington!
So it was working as expected, because nobody wants to see UK citizens getting accustomed to French cuisine and coming back only to complain about beans & fish!
I was once refused check in to fly to NZ from the UK because their system said I needed the Australian transit visa - UK-Schiphol-Singapore-Sydney-Christchurch.
It eventually turned out that I did not actually need the transit visa as I did not meet any of the requirements - time in airport, leaving the airport or no ongoing flight. This was only clarified after the holiday.
I was actually eligible to transit without visa (TWOV), muppets.
They had to rebook my flight using the Singapore - Auckland - Christchurch route for the out bound legs, but they had forgotten to rebook me on the return legs.
I found this out when they refused check in again a month later.
Cue a 24 hour delay and extra expense for car and accommodation
After a very stern email of complaint to the airline, I did get my extra expenses repaid, rebooking fee refunded and enough compensation to pay for part of the holiday.
Almost as bad the the time my credit card company did not clear the "hold" for the first 2 hotel stays that covered the first 3 weeks of the holiday, in effect double charging my account. I could not check in to the last hotel of the holiday using my main credit card and had to break out the "break glass" card. Cue another complaint and all currency conversion charges were waived for the months holiday.
If you are told to stand down by management then stand down
Continuing to work and fix the problem will not get you the thanks and praise you're expecting. As in this case being told to stand down will be because somebody else is working on a fix. There may be several reasons why this is the case but there are two basic ones; Somebody else is to blame for the outage so they have been told to fix it; Or somebody else (usually more senior) has volunteered to fix the problem cos they want the kudos. In either case if you fix the problem you'll be in trouble for one simple reason, the somebody else will be left with egg on their face and they will therefore ensure that you get in trouble.
Not only that but you've also embarrassed management. They made a decision and by fixing the problem you have called that decision into question. So management will be out for your blood too.
Or number 3: The problem is not as simple as it appears to you, and whilst you are busy fixing the bit of the problem you can see, the rest of the iceberg is going unfixed, with the potential result that your "fix" actually makes things worse.
The alarm bell that should have been ringing in everyone's head with this anecdote is why "copying some files" fixed the problem? Was the problem actually missing files, or was it something else that was looking for those files when it should not have been doing so? Did dumping those files into a folder on the user's PC actually mask an underlying problem, possibly of a more serious nature (such as a security flaw)?
It sounds like it could well be a "fix the symptom and not the problem" solution, where the problem has not been properly understood, which is the very worst kind of fix.
In a past life, I've been unfortunate to have worked with stuff where Crapita is involved. If anything, this makes it all the more likely that fixing the symptom is only going to exacerbate the underlying problem.
It's the difference between working out why your code in a financial system is out by a penny and fixing the root cause, and just adding a penny to the result. It might fix the output, but is both wrong and potentially dangerous.
The correct way to handle it is to tell either the crew that is on the case, or their manager, what you happened to find when you were messing about on your own time.
Let them decide if it's a fix, or tempporary patch, or just a wild shot in the dark.
Also, as this was during the slow transition from XP to Win7, it's possible they had already decided that they would use this to get he stragglers to finally update!
It was made very clear to us[0] that you'd only get a 2:2 (ie 50%) if you just regurgitated what the lecturers told you. I did an Erasmus year and it was a shock to see that in France you're marked on how much you recall.
[0] English uni in the late 90s, stuff may have changed!
It probably depends on the subject. The exam questions on the undergraduate degree I did (a physical sciences one) were based 100% on what we had been taught in lectures and workshops. I would imagine most scientific disciplines (such as medicine) would be the same - they are a test of whether you have learned the information you have been taught, not extrapolated from it.
I guess there might be room for learning beyond what the lecturers were teaching, but I'd take that as a sign that the lecturers weren't doing their job fully.
For me, we had part of our grade based on lab assessments, and part on exams. The exams would contain questions like, "show how the electrons move during a Diels-Alder addition," "what is the missing step in this well-known reaction, " or "which symmetry group is this organometallic compound in." All the sort of thing which have one definite correct answer (possibly with the exception of the Diels-Alder one, but I'm pretty sure either mechanism would be marked right, because things like temperature and pressure might influence the exact mechanism that takes place).
Of course, university learning isn't lectures alone, there were plenty of tutorials, workshops, and practical labs, but, again, this sort of thing is going to depend heavily on what subject you are reading, and which university you do it at.
does not lead to riches and rewards......
What it does lead to is being expected to fix stuff as well as your normal job... and indeed filling in for 3 other people who should be fixing stuff, but due to their inability , are fixing burgers/unemployed...
Oh and it is never a typo when crapita is spelt crapita.
This post has been deleted by its author
When I was working at Scientific Generics I was contracting doing E1 line comms stuff, anyway I needed to look at the PBX that also had 2 E1 lines that came into the building.
While looking at the setup I noticed a timeout to do with dialing an outside number. After working there for some years I had been quite irritated, when dialing an outside line if you looked down at the phone number for a split second, it would time you out and you'd have to start again.
So - I changed the timeout from 1000ms to 3000ms.
It worked great when I got back to my desk! No more getting cut off mid-dial!
My mistake was announcing the change to the company by email. I didn't get the bollocking, but the Facilities Management guy did!
I never found out who was annoyed. But it was never changed back even a few years later.
As much as everyone wants to direct their angst at Craptia, I have a little different perspective.
As someone that is in charge of IT security for a mid-sized company, I would have had "Terry" written up for this. He took company PC images home (he wasn't authorized, I'm sure), and put them on his own personal systems. Then he placed VPN configuration files that may have contained sensitive information on a public facing website under a domain that he controlled that had the companies trademarked name as part of it.
Has no one read about how the LastPass breach happened? This kind of shadow IT is where breaches start. Those of us in infosec put policies in place for a reason, and we don't need a help-desk person going rouge like this.
And as a former service desk tech working in the NHS for 20+ years, sometimes bending or even ignoring policies completely is the only way to keep people alive.
One thing I did take issue with however was when the medical director of a hospital trust demanded that service desk staff create ad-hoc accounts (with full access to clinical systems and information) out of hours for agency staff who were brought into A&E at zero notice to cover sickness etc. This was to be done without question and without any checks being made or the usual required paperwork being supplied.
When it was mentioned that these agency staff could be anyone, such as potentially a journalist or someone nefarious attempting to gain access to said clinical systems, the demand from the medical director was that under no circumstances should the Information Governance manager be made aware of these actions.
Still feel safe, your lives in their hands etc?
doesn't surprise me. Some of the total c u next tuesdays who work in the NHS have to be seen to be believed. My poor wife has to deal with these feckwits nearly daily, she works for a large equipment supplier to the NHS and the level of basic intelligence and sh1thousery that goes on drives her nuts
I once worked for Crapita for a short time on a large UK government contract, after being made redundant by the same government department a week earlier.
After a couple of weeks I was in a Capita internal meeting and was asked a question , to which I gave an honest answer. After the meeting my boss bollocked me for telling "the business" what I did. What do you mean "the business" I asked - aren't we all Capita?
It turns out that two different departments from Capita were on the project - we were IT and they were Business support, and I was told that under no circumstances was I allowed to give "the business" information like that!
I left several weeks later.
I find this exact situation very unsurprising. What really grates my goat, is that the fix was probably published in Cisco's support database at the time this event happened. I personally logged the fix with Cisco during beta testing of Win 7 in 2008, and I know I was not alone in that. So how the hell Capita didn't know about it BEFORE the upgrade started is beyond me. But this is Capita, after all...
However, he was a complete idiot for publishing his solution on an external website, and including the VPN config files; for just about any hacker with brute force knowledge and the right tools, to download...
Surprised he wasn't sacked.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
