Salesperson's tech dream delivered by ill-equipped consultant who charged for the inevitable fix

Before you shell out big bucks to engage the services of consultants, perhaps consider this week’s instalment of Who, Me? and the adventures of a reader we’ll call “Norman” for the duration of this tale. Towards the end of the 2000s, Norman left an in-house software development “join the ranks of The Consultants.” “My tech …

[Anonymous Coward]

I've never made a customer pay for my mistakes. But I sure as hell have made them pay for theirs (especially where said customers have been warned well in advance).

The best time was when I was in a meeting explaining that a system was unstable and needed to be migrated from ASAP. Said system had been running for a long time, was unsupported, and had been flagged as an issue for a very long time. Management scoffed and said it was fine. In the middle of the meeting phones started ringing. Guess which system had just fallen in a heap and was now causing a massive outage!

[Anonymous Coward]

collapsed email system

During the 00s, one of my company's site was running emails on a completely obsolete HW system, on HP-UX.

Those days, "HW obsolete and unsupported" meant the stuff was WAY too old !

The 100 something users were peaking this system to a constant 100% RAM/CPU usage on a week day.

I urged them to upgrade to no avail (no budget, mate !).

It continued to run OK for a couple of years, they added users, there were more traffic etc ...

One day, it totally collapsed. I swear, I never saw this prior or after. HW was fine, OS patched and OK, the email layer as well.

There was just, not enough power to run the mail system at all vs. the users population. There were no HW upgrade path.

End of the day, ofc, they changed the system, since none of the users could do email for one month.

But it was the only time in my carrer I've seen a system collapse like this.

[Doctor Syntax]

Re: collapsed email system

Did it happen after anOS update? I've seen this happen on HP-UX. Again, I'd said we needed more memory prior to this happening. Fortunately there were spare slots available for use when my prieviously idnored advice was acted on PDQ.

[Anonymous Coward]

Re: collapsed email system

"Did it happen after anOS update?"

No, we had exhausted all types of upgrade by then: OS, RAM, CPU.

Only a new system would have saved the mail system.

[Jou (Mxyzptlk)]

Re: collapsed email system

Well, a 1750 REU can indeed be considered a limit, since those with more were not stable enough for production and therefore not supported.

[lglethal]

I'm sorry but I dont believe Norman could ever have been a real consultant, or perhaps I'll concede this being his first consultanting position, he just hadnt developed the necessary "skills" yet.

Every Consultant I've ever met has lived by the motto:

"Consultancy - If you're not part of the Solution, there's money to be made prolonging the Problem."

(motto stolen from despair.com, but really does seem to apply to a LOT of consultants I've met...)

[OhForF']

The give-away that Norman is not a consultant for me was:

Norman wrote the required code

Consultants in my experience provide a ton of documents but do not touch the implementation part.

If something doesn't work they'll find a document that can be interpreted that something was wrong/missing in the implementation. The documents contain enough weasel words so you can't point out that different documents contradict each other.

Once i figure out how to get the big bucks but not being responsible for anything I'll probably call myself a consultant as well.

[Killfalcon]

For a few years, half my job was re-writing stuff made by consultants who weren't exactly beyond contact, but that management had a strong policy to never contact again.

On the one hand, it is always easier to rebuild than design from scratch, but on the other hand - you should not build seventeen nearly-identical modules for handling client surnames (based on which broker submitted the business).

Found that one out when one broker file would not go through, because while it was fine disambiguation "Mr & Mrs Smith" it choked on "Mrs & Mr Smith".

[Outski]

I got a reprimand many years ago for saying I wouldn't take on a mail migration project. What the potential client had was working well, might have needed a tweak or two, but otherwise, fine.

So they pay us 2 m ringgit, and what have they got? Email and calendar, just like they had before.

Icon for anyone who migrates an email system just because the old system is unfashionable (and they'll be gone before the migration's finished).

[Arthur the cat]

a sharp new suit and a misplaced sense of confidence and superiority

An excellent summary of consultants.

[I Am Spartacus]

Re: a sharp new suit and a misplaced sense of confidence and superiority

I have always thought that a consultant is someone who, when you ask him the time he takes your watch, tells you the time and keeps the watch.

[MrBanana]

Re: a sharp new suit and a misplaced sense of confidence and superiority

In all my job roles, I have railed at the title "consultant". It's actually a fair description of some of my core competency, but I would never want that name attached to me.

[Ikoth]

It's all the same, innit?

In the early 2000's I did a stint as a "Systems Engineer" for a large reseller / integrator. We SEs were a chargeable resource with the responsibility of installing & configuring whatever the sales guy could flog. In other organisations we'd have had the title "Consultant", but I guess that title would have commanded a higher salary.

At the time, their sales crew were the very definition of "coin operated" - if there was cash on offer they were all over it like a rash, and obviously selling professional services was very lucrative indeed. The SEs were allocated to a particular sales region with a mix of abilities and specialisms with each team. With a few notable exceptions, most of the sales guys didn't have a clue about what they were selling - one had been selling fleet verhicles in his previous job.

This caused no end of problems as by and large they had no understanding of the skills needed to deliver a particular project. Culminating in me, a fully paid up SQL / IIS MCSE, coming back from holiday to find I'd been allocated as the tech lead on a country wide Unix / Oracle roll out. When I challenged the offending sales guy he uttered the never to be forgotten line - "it's just servers and a databases, you'll be fine".

[Jou (Mxyzptlk)]

Re: It's all the same, innit?

> "it's just

A classical line to go one step beyond and challenge a quit over a sales guy who does not know the company. And if you don't "win": Quit stating the exact reason. For you it is a "Win-Win". For them: Who cares!

There are sales people out there actually doing their job well, but the incompetent type must either be weeded out, or left alone.

[Alan Brown]

Re: It's all the same, innit?

The incompetent type usually manage to weasel their way into upper management AND make everything someone else's problem

The classic one is to sell the inoperable, then when things come to a head hold a meeting, goad the absolutely incandescent customer into swearing and then stand up & walk out complaining you can't work with this hostility

[cheb]

Will next week's Who, Me? be about a journo who forgot to add a couple of important words at the start of the article?

[RockBurner]

I doubt it mate.... consultants never can be found again once a mistake has been discovered and made public.

[Caver_Dave]

Sold for the wrong job

Back before the turn of the century I developed the hardware and software and supported the first in-car telemetry system for F1, and lap timing systems for use by a multitude of formulas (including Nigel Mansell becoming a reseller when he moved to the states.)

I received a call from someone at one of the largest multi-national motor manufacturing companies (who also did some racing) to come down and have an expenses paid chat about an opportunity. It was all very clandestine, but not that unusual in the industry.

I was shown into a darkened room with a spot light on a chair. With trepidation I sat down, with the immediate thought that this was a rouse and I had cocked up on a defense contract of something! As my eyes became accustomed to the darkness I became aware of a row of tables down the length of the room that I was facing, and then soon after the 10 (ten!) people sitting in silence until I said good morning. I then suffered 20 minutes of being bombarded with questions regarding engine timing systems. I knew enough to answer the simple-ish questions, but when they asked about the pin-out of an XYZ Lucas connector then I had admit defeat and ask them what this was all about.

It turned out that these were the Executives and most Senior Engineers of the company and I was interviewing for the top position in their Engine Management Division.

I made my excuses and left.

It turned out that the HR person who organised everything had confused my "F1 lap timing" with "F1 engine management timing". They were sacked and as they could not sign the expenses claim, I would not receive that either.

I put it down to a learning opportunity, but was miffed at losing a day holiday to attend the fiasco.

[Tom 38]

Re: Sold for the wrong job

During my first year of uni, I wanted to get some commercial experience, and as luck would have it, there was a division of Crystal Reports operating in my home town (at that time called Seagate Crystal Reports, which dates this). They were advertising for a full time position, so I sent them my year 1 undergraduate CV and a lovely letter asking if I could join them as an intern for the summer.

They replied back, asking me for an 9am onsite, which meant an early (expensive) train down to London and then back out to my home town, and they assured me they couldn't do it any other time and they'd pay my expenses.

I turned up, met HR and they immediately comped me cash for the train tickets and told me this would be about a half day of interviews and technical assessment. First technical interview: So, when do you graduate? "...err - in about 30 months time?".

The interviews ended at 9:35.

[Caver_Dave]

Now quickest interview ...

A major global cosmetic firm were offering 3 times my salary at my first job (mentioned above), to basically write db filters.

Did the IBM IQ test administered by the UK IT Manager's Secretary. After 30 minutes of the allotted 2 hours, she came in and asked if I wanted a drink and I asked if she wanted the finished paper. I sat around for the next 1 1/2 hours, (without the cup of Tea!), before being led to the IT Managers Office Door.

He didn't even look up but said, "You will be bored here" at which point the Secretary closed the door and led me out. I didn't even get to say "Hello" to him.

[Sam not the Viking]

Re: Now quickest interview ...

In one of my periods of being disgruntled with my then employer, I applied for a job with a firm of consultants. I knew the firm from previous interactions and whilst their history was superb with some excellent completed-contracts under their belt, the new mob were 'branching out' into realms that their expertise was not suited (think small jobs needing small overheads). Nevertheless, before the interview proper, I had to complete a psychometric assessment and was given two hours for completion. After fifteen minutes, it was done and the astonished HR manager reckoned this was a record and offered me a job without further review.

After consideration, I rejected it, due to panic from my existing employer who gave me an immediate bonus then a huge upgrade in salary, but also because I reckoned the consultants were desperate to employ anyone. My subsequent dealings with them confirmed my assessment.

[JHC_97]

Re: Now quickest interview ...

I got one of those for of all things a java job in 1999. It was based on assembly memory management. A vital skill in java :-). I'd never done assembly my comment to the manager interviewing me well that was tough. Nevermind he said most don't pass. Oh i said i didn't say i got anything wrong i said it was tough.

[Outski]

Re: Now quickest interview ...

I was going for an interview at, I think, Anderson Consullting, before they became Accenture, as a Notes Developer.

Now, as most people here should know, AppDev in Notes is not like traditional programming, yet they still insisted on an exam about what I knew about memory pointers, .dlls and the like.

I walked out, saying we'd all wasted our time, and then called my agent to say sort out what you send me to.

Later that day, I had another interview with a company I stayed with for 15 years, even after spilling my coffee all over the table (thanks for the hereditary tremor, Dad).

[Martin]

Re: Now quickest interview ...

My two quickest interviews were when I was a contractor.

The first was when I arrived, and the guy thanked me for coming, and said "So, first question - where do you see yourself in five years time?" I was a bit surprised, and said "Well, I don't know, but I don't expect it to be here..." It rapidly transpired that they didn't need a contractor, they were looking for a permie. The guy apologised profusely, and paid my expenses. The agent who sent me along got an earful from me, and no doubt got an earful from the company too.

The other one was even quicker. I didn't do defence work - it clearly stated that at the very top of my CV. An agent phoned me, and asked me to interview at Decca Data Systems. I got a confirmatory letter (long ago, before email!), which fortunately arrived on the morning of the interview - which was when I discovered the company was in fact Decca Defence Systems. So I phoned the guy who was expecting to interview me, apologised and told him I wasn't going to be coming for a job I clearly wasn't going to accept. He was very nice about it. Unlike the agent, who had the temerity to ask me why I hadn't gone to the interview.

I said - because I don't do defence work.

She said - Why not?

I said - Not that it's any of your business, but it's a matter of principle.

She said - Well, that's all very well in principle.

I said - It's worked out fine for me in practice too, and put the phone down on her. Didn't use that agent again.

[KittenHuffer]

Re: Now quickest interview ...

In the past I've had a similar 'agent' experience as yourself.

I had an agent describe a job to me (without telling me who the company were) and gave me a location that was 10 minutes drive away. When the interview confirmation arrived (the day before the interview) I discovered that the location was actually 5 miles (of hell) further down the road, and that the company were mobile phone contract resellers (who had a reputation only slightly higher than that of professional puppy kickers)!

When I phoned the agent to tell him it wasn't gonna happen he tried to tell me to 'Man up!' and go to the interview, at which point my relationship with that agent was permanently broken.

[Martin]

Re: Now quickest interview ...

OK - what did I say to deserve a thumbs-down? Is someone really affronted because I don't want to work in defence? (I wouldn't have worked for a tobacco company either, though that "opportunity" never came up.)

Anyway, retired now - don't have to worry about that any more.

[Jou (Mxyzptlk)]

Re: Now quickest interview ...

If you are retired you should have been long enough on the internet to know that there are always haters. 17 up and one down? That is a good statistic. The nuclear threads mess my statistic up since I have a very very German view on this.

[MrBanana]

Re: Now quickest interview ...

I was once sent to an interview where I had the necessary skill set in C programming and some kernel development knowledge. Second question, after would you like a cup of tea, was a very detailed examination regarding my knowledge of the internal processing of the nap() system call. I wasn't very familiar with that part of the Unix (SYS VR4 as it happens) kernel so could only make polite excuses about that one, very specific item. There was no third question. They clearly wanted to grab some information from a kernel expert about a very specific problem, and had no real intention of paying someone if they could get the answer from a hapless schmuck during a job interview for free.

[Outski]

Re: Now quickest interview ...

On the phone interview for my current job, I was asked what I'd look at for a Domino server that was really struggling. I mentioned a few things, and by the time I walked into the office, they'd taken my recommendations and sorted out half the problems, so not everyone's like that.

[Martin]

Re: Sold for the wrong job

Not your fault you'd lost a day - I would have thought the company should have reimbursed you anyway. I'd have made a lot of noise about it.

[Terry 6]

Not his fault surely

The customer wanted a system set up.

He set it up.

It did what they asked.

Then they decided to use it in a totally different way ( automated emails ) without thinking through how it worked and the implications of that change.

If there was any error, it would be in not making sure the parameters for the functionality were engraved in foot high letters on the wall of the server room.

Quote " ...But the automated process misconfigured the emails, so the required info was missing "

[ElReg!comments!Pierre]

Re: Not his fault surely

Well, the articles does state that the processing engine did not remove invalid mails aftr bouncing them, and thus re-processed old emails each time it ran, whic seems like a pretty major design flaw to me

[Alan Brown]

Re: Not his fault surely

Flaw or not, it would have ended up playing perpetual ping-pong regardless

It's not the first time I've seen a system loop itself into the ground on this kind of misconfiguration and most of the time they don't need an exploding mailbox to achieve it - that just makjes it happen faster

[agurney]

Tales from the future

Towards the end of the 2000s, Norman left...

Hopefully not near the end, we're only 22 years into the millennium.

[Aladdin Sane]

Re: Tales from the future

Wibbly wobbly, timey wimey.

[Doctor Syntax]

ISTM that the problem was caused by the business unit that implemented the new function in three respects. First, not meeting the requirement as to the format of the emails being sent, secondly mishandling bounce and finally, not testing before release.

[Dave314159ggggdffsdds]

Idk, I'd say that by the time it was written any auto-sending email system should have coped gracefully with the problem.

[doublelayer]

How would it do that if it was set to interpret any incoming email as a request and bounce those that weren't valid? To handle this, it would have to cache invalid requests and deduplicate identical ones, which would have stopped the loop after two of the autoreplies to invalid messages but wouldn't have accomplished any other goal, such as actually working or not filling the queues with bounced messages and autoreplies for each new incorrect request. It just wouldn't have gotten into an exponential loop, but things still wouldn't be fine. If anything, the system that should have handled things differently is the system that sent automatic replies to a bounced message, as that's easier to figure out from the headers.

[PM from Hell]

Both ends need to prevent autoresponse hell

I'm a PM and find time after time I have to remind developers that what we are implementing has to cope gracefully with crap, this can be bad api's mis-configured data, random text or spam emails.

There have been so many examples of email auto-responders locking up exchange servers that there is no excuse for not coding a solution to this. If there is no alternative the service should pause itself and raise an alert which can be sent to a mailbox and the ITSM service to alert service management. the alert should include a reference to a log file entry which captures some meaningful diagnostic, at lest enough to identify the sender and the individual message plus failure reason. Your development should be idiot proof and bomb proof, you must always assume that every other team is incompetent and will send you meaningless data at some point. That will build in some forgiveness when there is a business process change on your end which your code does not cope with (and was never designed to). None of this is new or clever.

[Alan Brown]

Re: Both ends need to prevent autoresponse hell

"There have been so many examples of email auto-responders locking up exchange servers "

Exchange's mail handling was so diabolically awful back in the 90s-00s that you only had to look at it sideways to achieve that

The devs had a passing acquiantance with RFCs - meaning they knew the things existed but hadn't actually read them and made a bunch of fatal invalid assumptions about header/body sizes (some of which persist to this day)

[xyz123]

I once saw a company using the "local update" functions built into windows update. Basically instead of every one of 20k PCs downloading an update from Microsoft, they would 'give' the update to other PCs on the LAN.

Well some smart guy had also configured the system to not allow data to be copied from PC to PC (it had to go to a network folder THEN the other PC could download).

This system had a blocking-ability. So if 2 PCs tried repeatedly sending data to each other, it would blacklist them on the network.......essentially considering one PC was trying to infect the other PC with something nasty.

Cue MS update...every PC starts asking every other PC if they have the update.....essentially the entire network blacklisted every single PC from itself.

took 5 days to fix (literally someone going around with a USB key with the fix to every individual PC), and the company lied to its clients claiming it was "an ISP issue".

[Anonymous Coward]

Ohh, the lie to clients bit. Been there, seen that.

When I left a previous job, a manager there could not contain his enthusiasm to get rid of everything I'd built (solid, reliable, but as he only understood some bits of Windows and this was all Linux & FOSS - "not something he understood"). I was informed by one of my ex colleagues that not only did he shut down servers, he pulled all the cables out and removed them from the racks.

And all was fine "for a while".

Most of the domains on the DNS primary had a timeout of a week. So after a week, all the public secondaries dropped the zones - including my ex-employer's own. Cue clients with broken system, etc, etc. And him (and his team) hurriedly entering domains into the cloud hosted system they insisted on using from an old dump I'd done for him of our DNS.

Of course, it was all an ISPs fault when the clients asked what had happened.

I'd consider it funny if it weren't for the many nice and honest clients it affected.

[Anonymous Coward]

One of the biggest problems that needs fixed to this day that has been brought about and perpetuated by most IT staff is the use of the word issue to describe a problem.

Issue does NOT mean problem, nor has it ever.

[Anonymous Coward]

Perhaps your father should have not given his issue to your mother?

[Dave314159ggggdffsdds]

From my days doing support, I used 'issue' a lot because often there wasn't an actual problem. Or at least no problem except a stupid user.

[Prst. V.Jeltz]

gitlab seems pretty stuck on using "Issue" to mean

"Future planned development enhancement"

[MisterHappy]

We don't have problems, we have "Opportunities"

Me: "We have the opportunity to see how long the data centre can run mid summer with no air-con."

Boss: "Huh? What are you talking about."

Me: "Well, as we never have problems, I can't say that the Air-Con is broken."

Boss: "Don't take the piss!"

[Alan Brown]

Re: We don't have problems, we have "Opportunities"

"I have a very severe drinking opportunity"

[Anonymous Coward]

Issues / Incident / Problem

All 3 terms have defined meanings in grown up IT environments.

An issue is anything which may need resolution for a project to achieve the next milestone. Its not something that may happen (that's a risk) its something which has happened and wither needs signing off as acceptable, a workaround being implemented of a config change / code fix being developed and tested.

Once in live operation and into the ITSM managed sphere then operational incidents are recorded to document reported failures / system behavior issues etc. these should be managed to resolution via the service desk. If an incident identifies an larger underlying fault with the system, or is being reported by a large number of users then a problem is raised and the individual incidents are attached to this. This is then managed by the ITSM Problem / Change manager who will co-ordinate the support / development and vendor resources required to deliver, test and implement a fix.

[disgruntled yank]

Ick

I worked for a government contractor, on a project managing DG minis running WordPerfect Office. There was no mechanism to do out-of-office email forwarding, so one of our guys wrote one--in COBOL, as it happened. He did not include any loop detection, and the customers (or we) were careless about turning the OOO settings off, so periodically a few in-boxes would fill up. It seems to me that I tried (despite near-total ignorance of COBOL) to add some loop prevention logic, but it has been quite a while.

[Filippo]

"Consultant" has become such a loaded word, that I prefer to be called "Mercenary".

[Mostly Irrelevant]

Autoresponder tennis is something I know all too well. One year when I was in university I set up an auto-response when I left school for the summer and when I returned in September found I had many thousands of emails. I had been sent at least one email from the school with an invalid email address that returned a "invalid address" message from the email server which then elicited an auto-reply, repeat forever. This was particularly stupid because both actors where the same email server.

The webmail system the school used was only capable of deleting emails it was displaying and it could only display 50 at a time so I had a choice, contact IT and wait forever to be ignored or just figure it out myself. First step was obviously to disable the autoresponder, then I had to figure out how to get rid of the emails. Eventually I found how to log in to the system via pop3 (no instructions given) and I wrote a program to connect to pop3 in indiscriminately delete said emails without downloading them first (my first attempt with outlook yielded a download progress bar that stretched out into infinity). It took about half an hour, but I was free after that.

[Prst. V.Jeltz]

Tsk!

Towards the end of the 2000s, Norman left an in-house software development “join the ranks of The Consultants.”

Sentence #2 and the grammar is slipping already!

[herman]

Re: Tsk!

Engrish is to be helping and not to be laughful at.

[ICam]

Re: Tsk!

Sadly, it's an increasingly common occurrence at El Reg these days.

It would seem that some of the staff are now so slipshod with their articles they don't even bother to spell check them before publication, never mind grammar checking them.

[SImon Hobson]

Ah, the joys of email tennis - note, not quite autoresponder tennis.

Maaany years ago, I got my employer into email - back when this meant dial up (and POP and SMTP between clients and server), but we did have the heady speeds of 64kbps over ISDN. Initially we used a piece of software running on an old Mac, but after a while we started to outgrow it. So we setup a different server, and started migrating users - as this meant changing user settings, we did it one user at a time (there weren't many), setting an auto-forward from the old server to the new one for each user just before we changed their settings.

Then one day, email "slowed down somewhat". Both systems had the same email size limit - and someone had sent an email, to a migrated user, that was just under the limit. So when it got forwarded, it had grown to just over the limit and got bounced - but not before almost all of it had been transferred. IIRC, the bounce message included the original message - and this was also over the limit.

So the two servers were happily bouncing these large messages - and IIRC filling up the disk on the original server. Oh those heady days when 80M was huge !