Reply to post: Another consultant take of woe.

Heard the one where the boss calls in an Oracle consultant who couldn't fix the database?

Anonymous Coward
Anonymous Coward

Another consultant take of woe.

A number of years ago a company I worked for hired a new ops manager. He was one of those who didn’t trust his staff’s technical skills and preferred spending company cash on external consultants who told him the same the internal staff would have told him.

The company ran a number of critical databases on IBM Power boxes and a consultant was brought in to “health check” them. The guy who built the systems had built a redundant solution with multiple VIO servers managing the SAN and LAN traffic. He had gone on holiday when the consultant rocked up.

The manager had told the support team to give the consultant access to the VIO servers and all the LPARs on the P series boxes. After a poke round, the consultant started running a check script on the VIO servers. The first server kicked him out of his SSH session once the script started so he went on to the next VIO server. The SSH session was kicked out here as well. In fact, every time he ran his script, he was kicked out of his SSH session. More worryingly he couldn’t get back in. At that point he said the scripts would take a while to run and this was expected behaviour and he then left for the night.

Soon after that, operations started getting alerts, the application servers were struggling to get to databases. None of the database monitors were returning any information and DBAs couldn’t log into the systems. To all intents, all the databases were down and nobody could get into the system.

Frantic calls were made to the guy on holiday who built the system and despite his best efforts on the phone, nobody could get the boxes back online. He actually drove 4 hours back to the office and managed to sort the problem in about 30 minutes. The script the consultant ran caused a kernel panic on the VIO server, this was a documented issue the consultant should have known about. The issue was compounded by him causing a panic on every VIO server and not checking why his SSH session was terminated.

At the post incident review, there was a concerted effort from both the manager and consultant to shift the blame for the issue. The manager lasted a few weeks longer and the consultant was left in no doubt as to how his invoice would be dealt with.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon