DataSolve - Sunbury?
Remember using their Mainframe bureau in Sunbury
9 posts • joined 11 Jan 2019
Remember British Satellite Broadcasting (BSB) and their Squarerial (The square shaped satellite dish), a long since merged/defuncted UK corporation was using a VAX 11/785 in the development of this failed product (This is a whole story by itself in marketing pushing bizarre requirements to an engineering team) - The VAX 11/785 had been taken off contract as they had a lot of bright engineers and they would self maintain to save some money. So all seems quiet/good for about 8 months and we get the call, can we come to site the VAX 11/785 is borked and they need assistance (Which we will charge on a T&M basis). I arrive on site ask for some background as to what has happened, get basically a story that everything was good they had a problem but this problem quickly moved to a complete system failure. They had done some diagnostics but not touched the VAX physically.
The VAX 11/785 was switched off (Weird, not normally switched off, even when a fault is present) - Asked why it was switched off - They said they could not start the machine - The VAX 11/785 had a cutdown PDP that was the boot controller, so you would normally have this active even when the main machine had major issues.
So, my first action is to open the main cabinet door to look at the logic boards (The VAX 11/785 CPU was basically a set of logic boards connected to a backplane) - Opening the cover to the logic boards I asked again if they had done any "hands-on" troubleshooting - They said no, no-one had touched the machine at all.
So I said that was strange as one of the Logic Boards was inserted the wrong way round (Top was on the bottom rail) - This would have taken a lot of effort to do as the boards were keyed to go in easily one way only (i.e. with top of logic board aligned to the top).
Cue a lot of people suddenly looking guilty - They fessed up that they had taken the cards out to reseat them as they thought that may resolve the problem, they had a problem with one card being difficult to get back in, which in fairness was an issue with this enclosure - They thought the card was in the wrong way so flipped it an re-inserted and with what I can only assume was superhuman strength managed to get it all the way in.
It took some effort to prise the card out and peering at the backplane revealed crushed pins on the backplane.
So 2 days later, new backplane, assorted new logic boards and north of £35,000 they were back in business (let alone the business cost of not having the system up and running). They basically they had to spent about 8 years worth of the maintenance contract that they had cancelled in one go.
Their management did try to say that it was easy to make the mistake and it was our fault that this was possible (Negating what I can only assume was obvious visible guides not to do what they did, but the sheer brute force needed to do it). But sanity did prevail and they paid the bill and even put the system back on maintenance contract.
Alas th Squareial did not help the company make the expected millions (Especially as told to me by one of the lead engineers the spec they were instructed to build it to did not factor in cloud cover .... So in testing/demo phase all was good until it got cloudy - so expense re-design required) - And BSB got bought by Sky to become British Sky Broadcasting and the Squareial was consigned to the dustbin of "what were they thinking".
Circa 1988 and the Wang Laboratories was rolling on globally their new hardware maintenance service they sold to their customers, I good friend of mine was trying extremely hard to get the name changed for the UK market as selling the WANG CARE service he thought might be difficult. He was of course ignored by the US based marketing team and all hell broke when released in the UK with a full advertising push etc.
Circa 2004 in Texas, there was an advanced training program - Technicians With Advanced Training - You could become a certified T.W.A.T - As the only Brit and seemly only person who spotted the issue (and kept very quiet), it was marvellous when this was rolled out beyond Texas ......
It's the early-mid 1990's we are an outsource supplier to a very large UK bank stock brokering operation - They have invested heavily in a clustered VAX infrastructure that has eliminated most of the single points of failure, except one big one in that the cluster (And shadowed disks) were in the same physical data centre and in fact in the same large bank customer only data center hall/room.
This clustered VAX infrastructure worked extremely well and the bank was very happy with the high availability, much improved speed and the service we were delivering, so much so, they wanted to bring a bunch of their senior executives to our facility to show them what the banks money had been spent on.
Cue to day of the visit, everything was going really well, the final part was an escorted visit to the actual data room where all their kit was housed - Some questions asked by executives but suddenly the chairman announced that he understood that it was impossible to shutdown the computers (Great selling of high availability by the account manager) and suddenly the chairman reached out to the big red button on the wall. Literally mill-seconds from disaster I managed to intercept the chairman's had (with quite a bit of force) and stop him hitting the EPO.
However the aftermath was me apologising for the "assault", the bank chairman thinking we had something to hide and overall 12 months to rebuild the customer confidence in us - Even though we did stop a complete powerdown which would have cost them hundreds of thousands.
Lesson learnt, never ever take customers into the data center even if they own it - So next step was a couple of windows installed so customers could see but not touch.
Military Ship Radars are very powerful things, they should never be activated in port and being anywhere near the main array when active requires some serious lead underpants. Circa 1988 HMS Invincible is in Devonport Dockyard, the main radar array was accidentally activated for 2-4 full sweeps, at the time I worked for a Mini Computer company as a systems engineer, almost everything in line of sight that got a dose of the HMS Invincible radar sweep either crashed or had memory corruption. A very busy day, initially confused as to the cause of the issue, but as first system call was from Devonport Dockyard and the Radar Array activation was being talked about it didn't take long to connect the two events (And all the other calls coming in). I seem to recall there was some very serious disciplinary action taken on a number of the naval crew from HMS Invincible, it was a very serious matter and could have been worse for the crew and people nearby.
Biting the hand that feeds IT © 1998–2020