Posts by Barking House

Techie saved the day and was then criticized for the fix

Floor loading, what is that

When I was a field engineer I visited a large Telco DC to be greeted with various people lifting floor tiles looking for some specific cabling - I immediately advised the customer contact who was escorting me that there seemed to be too many tiles up and the floor might be in danger of collapse. I made quite a fuss as I thought their people might be in some real danger of injury and they did yank everyone out of the room. Whilst the senior DC manager was sought for his opinion part of the floor collapsed tipping a VAX Mini onto its front (Though impressively it kept running).

In the aftermath I was blamed as the customer said if their people had been in the room they would have been able to prevent the floor collapse - They asked my company not to sent me to site in the future as I was banned ....

PS: The ban only last about 2 months until the Senior DC owner was moved onwards to a new position, this was not the first major incident I found out that had happened under his stewardship.

Ad agency boss owned two Ferraris but wouldn't buy a real server

Re: Aaron?

Classic Judge Dredd - The Judge Cal saga

Developer's default setting created turbulence in the flight simulator

Military IT Moves

From the 1980's, watched opened mouthed as Royal Navy matelots manually dragged a PDP system out of the onshore DC I was about to enter and loft it onto a flatbed truck. They tied it down, put a canvas over it and set off. I suspect it was not functional when it arrived and was offloaded to whatever destination it was going to.

I did raise my concerns with the person who was hosting me on site but they just shrugged their shoulders and said it was not part of their IT domain.

Owner of 'magic spreadsheet' tried to stay in the Lotus position until forced to Excel

Mission Critical, even when switched off

I had a major Telco asked for assistance on a server consolidation. The biggest challenge was the kit that was mission critical but either had no owner in the organisation or an owner who basically stated that it was so important the entire fate of the company rested on that particular server. We took the approach of consolidating all the servers for which we had reconciled Owner, application function, business alignment. We then started to switch off (Physically) the servers we could not identify the owner, the real business requirement etc.

We had a couple of servers that did pop up as a requirement (They were important on a quarterly basis) and get these assigned with new owners etc etc.

we went back to any owners who had stated the mission critical nature of their servers 14 months in to advise the servers would be decommissioned and removed - One guy literally screamed that his server was so important that the company would be out of business in a matter of weeks if we did this to his server - I was somewhat perplexed as the server has been switched of for over 14 months at this stage and when we advised him of this fact, he seemed some what deflated and meekly said he would agree to the server being decommissioned.

The switch off approach did work, there were some servers used on an infrequent basis but we caught most of these early on and the vast majority that were switched off were never switched back on and there was very little issues caused (Just some minor inconveniences).

PC tech turns doctor to diagnose PC's constant crashes as a case of arthritis

South don't work in the North

In the 1980's it was common to see very large (Heavy) Video Terminals - The ones used for graphics were particular expensive, large and complex. I get a call out to a customer who has 3 new 29 Inch colour VT monitors and he is unhappy as they are using them to display technical drawings (Early CAD stuff) and the lines are "Wonky" on the side of the screens on all 3 monitors.

These are new monitors, I do a check on one to check if there is any damage due to shipping and to check the calibration - All looks good - Checking the serial number with our service DB I was informed these monitors were ONLY intended to be sold and used in Southern Hemisphere and would not be supported in Northern Hemisphere.

Turns out a lot of monitors back in the day were calibrated for the southern or northern hemisphere use.

Customer had acquired these as they were going cheap (Comparatively speaking) - Was not happy when told we could not resolve as this was a factory calibration and we had no way of fixing in the field.

I think in the end he traded them in for the correct hemisphere ones and paid nearer full price .........

To make this computer work, users had to press a button. Why didn't it work? Guess

Re: The human race could be safe after all

That is probably a good thing .......

We have redundancy, we have batteries, what could possibly go wrong?

UPS Failure ....

Worked for an a managed service provider and had a suitably impressive DC with lots of redundancy, all top notch, including diesel generator that was managed by the local power grid provider (Who used it to smooth out power supplies when demand was high). All great stuff.

However the UPS batteries as you will be aware do need some tender care and attention to keep them in great condition, this was done on a regular basis but unknown to everyone is that we have a defective battery (That past inspection) and during peak daytime running suddenly failed and produced an impressive amount of smoke, this triggered the fire system which duly shutdown the DC. Had a bit of fun trying to stop the local fire brigade from axing their way through the doors, we managed to show them the doors were unlocked (As part of the fire system released all locked doors when activated).

The damage was limited to the UPS battery and some associated kit, nothing dramatic, but then the fun started with explaining why the UPS/Generator et al did not keep the DC up - It would seem obvious that in the event of a fire that the DC power would be shutdown, but this seemed to be a surprise to senior leadership and a number of the customers .......

IT blamed after HR forgets to install sockets in new office

New DC, we don't need your help ....

A global major player in cloud computing engaged me to building their first internet DC, tough job, but it went very well. The second internet DC they wanted me onboard but they would do the design etc and I would be there to help with the testing strategy and such like. It was made very clear to me that I was there against the wishes of the Project Team as they did not need my assistance.

There were a number of items I raised as the project started that I knew need some adjustment but was very firmly told not to get involved and focus on the Test Strategy that I was being asked to do.

I asked if I could see the DC plans, the infrastructure architecture so I could start to thinking about the most appropriate testing strategy.

I raised one concern with regards to the cabling inside the DC as it was not clear when this was being done on the project schedule and what cables and suppliers were being used - Got a response that they would send the details to me soon. About 3 weeks later as the equipment started to arrive I raised the issue again, but this time it was evident that they had overlooked the cabling requirements !!!

I advised that this would have a significant effect on their schedule, they seemed to think it would only take a call (Using their name) and magically a structured cabling plan would be generated along with the cables and cables installed etc.

Suffice to say this is not what happened and initially fingers pointed at me, but I reminded the customer that they had taken full responsibility for the DC and infrastructure design ....

The end result for me was that the customer project team got brutalised and I was asked to take on a consultant role and basically drive the remediation and the rest of the schedule, I re-assigned the project lead and the acolyte who caused the problems (As the structured cabling was not the only major issue)

Nvidia promises British authorities it won’t strong Arm rivals after proposed merger

I see what you did there - StrongArm

When ARM was still British owned and still building its momentum there was of course the partnership with DEC - StrongArm

https://en.wikipedia.org/wiki/StrongARM

How to destroy expensive test kit: What does that button do?

The more senior the person the more likely they will press a big red button

Early 1990's had a High Availability VaxCluster system with 4 nodes, this system was very robust and basically acted like a fault tolerant system with no unplanned downtime. The financial institution that owned the system was very proud of its record of not having service interruptions (As these were very costly, hundreds of thousands in a few hours type.) and for reasons that are lost to the midst of time the CIO wanted to show the board the very expensive cluster system that was making them more money and enhanced their customer service experience etc.

The CIO said how in this single facility they had duplicate systems, power, comms etc and they were investing in a second site to provide improve resilience etc, the Board Chairman seem to have taken the CIO at his word that nothing would take the system down and wandered over to the big red emergency power off button and pressed it - This brought the tour to a immediate end, cost an eye watering amount of money in lost business, permanently ended DC tours (Even for the most senior of executives) and of course the Chairman was exonerated as how was he to know that the Big Red Emergency Power Off labelled button would actually do what it said.

But on the bright side, the CIO got promoted out of the company and the financial institution 2 years later replaced the VAX'es with a Pyramid Unix system (From Pyramid Technology which was eventually bought by Siemens) and although powerful and fast for the time the Pyramid system was not super reliable and the service interruptions became almost a weekly event.

Try placing a pot plant directly above your CRT monitor – it really ties the desk together

Adding China to my monitor

As a Field Engineer, I remember going to a call out to a factory that manufactured expensive pottery/china/plates etc and being presented with a PC Monitor that had been accidentally dropped into liquid clay. Was told it was not working and could I take a look and fix it !

They were most upset when I said it was beyond repair and as they were responsible for dropping in into the liquid clay they would have to buy a new one at their cost. The factory manager was most upset and said there was no point in having a maintenance contract if we would not fix problems.

All their kit on the factory floor and vast accumulations of clay/china dust, it caused all sorts of problems that we generally did fix but I always though if this stuff is killing PCs etc what is it doing to the people who work there !

Big red buttons and very bad language: A primer for life in the IT world

Re: RM-05s

I recall the service guide advising that the magnets used in the device (Motor) were so strong they would effectively destroy any wrist watch they came into contact with them !

Re: RM05s

At least it wasn't a RP04 which was a Sperry & DEC designed and manufactured beast, from memory it was about a size of a classic mini and had a capacity of 88MB's and you could feel the vibrations on the floor when it was working at full pelt.

Re: Replacement hardware?

My personal experience was that aerospace companies tended to keep what they had running for as long as possible because their product developments times were so long, they liked to have the same IT equipment throughout the development term (At least in the 70's, 80's and 90's). I recall PDP-84 having SCSI disks so this was definitely an option, but it would probably be like VAX's replacing RA82, RA90's with SCSI/DSSI disks it was definitely an option but if you had spent a small fortune on RA90's for instance people tended to want to get their money worth not to mention that the computer company would want to keep the nice high margin maintenance contract going

Re: Flashing leds on PDP-11

From memory, most PDP's had ancient LEDs (Think the ones with 2 prongs) but depending on the model, some (Certainly the older models) PDP's had an impressive array of traditional small bulbs and LED's which failed on a regular basis.

Vegas, baby! A Register reader gambles his software will beat the manual system

Metrics and pay drive behaviour

Metrics and pay drive behaviours, it is something I pay close attention too - Many times it was clear early on a customer engagement what they asked for is not what they wanted, the trick is to understand that early enough to reach an outcome that works for everyone. But also, I have been in a customer environment where it was clear that there would be significant resistance to any change and the corporate overlords used it as a good excuse to either terminate the staff or undertake a radical re-organisation ......

I haven't bought new pants for years, why do I have to keep buying new PCs?

So why isn't IT kit like Jet Engines ........

Circa 2009 and I was with a very senior leadership team of a large aero engines manufacturer, one of the questions they wanted to understand in more detail is why is it that they can design an engine that has an effective lifespan of many decades and yet the computers systems go obsolete much much quicker. The reason for the question is that they were maintaining all the systems used to design the engines and keeping them in place for the lifespan of the engines, this is so if there was an issue they had all the original software, data etc in place. The challenge being a 20-30 year old system (Think VAX, SPARC workstations etc) became increasingly expensive to maintain and fix (In fact for some systems they had to get some items they required via eBay). The conversation was difficult as they could not relate to the pace of change in IT against what they were doing in terms of development cycles etc - However we did introduce them to the world of Emulation and virtualisation - Impressed seeing a VAX system moved to a Intel Server running a VAX Emulator with a PCI to UniBus adapter and running about 10 times faster !

I built a shed once. How hard can a data centre be?

Re: A VAX8820

The cool feature was you could power on the VAX using the Pro350/380 via software only, no need to hit any power on buttons.

In the UK the naming conventions for building floors is Ground Floor, 1st Floor etc (So in UK the first floor is the 2nd floor if you are used to the US naming conventions). A small but significant difference in the use of language between the two countries.

Who knew that hosing a table with copious amounts of cubic metres would trip adult filters?

Early days of Firewall content scanning

This takes me back to circa 1995 and the early days of firewalls and content scanning for inappropriate words - We had a bank that cranked the very new firewall content scanning all the way up to 11. They were initially very pleased but soon reported that a significant portion of e-mails was not getting through. I looked at the firewall logs and soon discovered that the content scanning was looking for all traces of words that could be considered offensive, so for instance the word "digital" was triggering the rules as it detected the word "git" in digital. Hence a lot of e-mails getting blocked ....

Once rules were modified to be less aggressive the issue was resolved.

Let's... drawer a veil over why this laser printer would decide to stop working randomly

Never under estimate the ability of a user to out stupid you

Had a major corporation that was moving to using biometrics for user device logins (Finger prints login), in part to improve security, part to remove the users requirement to remember passwords which was problematic in this company and partly to demonstrate to their leadership that they were adopting high tech solution for security (!!).

The project itself for the design, implementation and deployment went pretty well and we started with a small group of high profile users first to test the technology (Before committing to a mass roll out). In this group was the CEO executive assistant (The one with real power :-) ).

First morning of the first day we get a call that the Exec Assistant could not login using finger print reader, a IT tech was duly dispatched - They said the finger print reader worked fine when the Exec Assistant tried it again when they got there.

Day 2, same issue reported by the Exec Assistant, same result when IT support etch went there, already got rumblings from Exec Assistant that she was not happy and thought this solution was wasting her time.

Reviewed other support calls and although some problems we understood the cause etc, no-one else had similar problem.

Day 3, same issue !!!!! It is at this point that I went to see Exec Assistant, to calm her down and to review the issue first hand. I was talking to the Exec Assistant about her morning routine to gain some insights into what was happening first thing in the morning - It was clear the CEO needed a lot of "looking after" and she would be in early and straight to work.

I asked the Exec Assistant (As it was January) did she wear gloves, she confirmed she did - She also confirmed that the first thing she did was login to her computer (Laptop - Which stayed in the office full time) and always did that before taking off her coat and gloves (So she could view messages etc - multi-tasking).

Being careful not to offend, suggested a slight change to her routine, gloves off first and then login with finger print - She had not connected the finger print requirement and having her bare finger being presented to the scanner, she had understood it just needed her finger to be "scanned".

A quick update to the training materials, guidance and FAQ's .......

Quick thinking and an explanation for everything – key CTO qualities

NASA, Mars and Sojourner ....

We had an important client coming to our large conference suite, it was an initial kick off meeting, the team were well rehearsed, we knew what we were going to present and everything was ready to go.

The sales guy was keen to show that we knew about the internet and kept asking if we could connect to the NASA Live Feed about the Sojourner Mars Rover (This was Circa 1997) - I said we could do that post the meeting but did not want to risk having a point in the presentation (We were using PowerPoint 97) that linked to an external internet page as PowerPoint 97 would sometimes bork itself.

10 minutes before the client came into the conference suite the sales guy had asked to make a change to the PPT, foolishly we let him .....

Que to client presentation, the massive BARCO projector was beaming in full VGA resolution and a new slide popped up with a link to www.nasa.com (Not nasa.gov) - Before we could stop the sales guy from clicking on it, up popped the National American Stripper Association (NASA) in all its glory !!

Suffice to say the meeting did not conclude as we expected and the Sales guy was in very hot water with his manager with pleads of how was I to know that www.nasa.com was not the same as www.nasa.gov

Unfortunately the really rather marvellous 1997 NASA.COM has long since gone and nasa.com has had some other websites since but none as inventive.

When you're On Call, only you can hear the silence of the clicks

NO Entry ......

I had in 1993 an analogue Vodac NEC P4 (Which had replaced a trusty pager) as I was told being reachable 24/7 whilst on call would improve service etc ......

Apart from the concern that this thing was literally going to microwave my head, it suffered from the issue of a lack of masts for service coverage where I lived at that time, so would have variable service connectivity. However one night at about 11:30pm I got a call from Ops team that the trading system had borked, the software house had made a change during the day and it seemed it had unintended consequences - The software house was of course unreachable, so I was off to site in my trusty RS Turbo Fiesta :-)

*Could not use my 9600 baud modem (Connected via a VT terminal) as this particular system was not cleared for remote access.

As I drove out of the town I lived in, the local plod switched on their blues and twos and stopped me - As they explained it looked suspicious that a relatively young man driving a expensive car at that time of night (!!) - Could I prove I owned the car, that it was insured and show my driving license ..... Of course all I had on me was my company ID badge, which was not cutting it with them nor that I was on-call and responding to an urgent call. At this point the mobile phone rang (It was in my jacket pocket), I answered it and with mobile phones still being expensive and relatively uncommon I was then viewed even more suspiciously as I had all but confirmed I was a drug dealer in their eyes.

On the phone was the ops team asking for an ETA as they new that if the overnight run was not fixed in the next hour or so it would be very very bad for everyone (As lots of money would have been lost etc). I asked the ops team to call the IT director and get him to phone me to help explain to the police I was who I said I was.

10 minutes of the police doing checks and finding out the car belonged to a leasing company (And not me directly) it was looking like I was off to the police station for a more detailed interview, my phone rang and it was the IT Director - Not known for being particularly knowledgeable on IT, but it seemed he played golf with the Chief Constable and told the local plod he was going to call him to get this issue resolved.

I was was given the go ahead to depart but with strict requirement to bring all documentation to the Police Station in the next 24 hours - I managed to get to site, fix the issue with the overnight run (Which was classic software developer stupidity).

But first time I was grateful that I had the mobile phone and not the pager, plus was able to add in an additional hour for the chat I had with plod :-)

There ain't no problem that can't be solved with the help of American horsepower – even yanking on a coax cable

Beware mystery cable runs

It is the late 1980's I am asked to visit a customer campus that is experiencing strange ethernet issues. I turn up at the campus that has two buildings that are separated by about 20 feet at their nearest points. I quickly confirm that they really do have an ethernet problem, to the point that is almost unusable. Do the normal troubleshooting questions, anything changed? / Is there any physical activity on site etc etc

Get a negative from the IT boys, so in the end it was up with the DC floor tiles and start to examine the transceivers (As the back bone was 10MB Thick Ethernet), found the transceivers and all looking good. But strangely the Ethernet cable seemed to go out of the DC wall. Ask the IT Manager where the cable is routed, he tells me that they dug a trench about 6 months ago to the other building to connect a small server room to the backbone. Had to convince them to get someone to dig up where the cable had been buried - Unbelievably they had just buried the cable in the ground and was looking very much the worse for wear !

Apart from the electrical issues (Electrical Potential issues) connecting two buildings with a cable which in itself could cause issues, the cable which was very much not intended to be buried in the ground outside and started to breakdown the outer coating which had caused the failure.

Advised them to get new cable, keep it in the DC and invest in proper connectivity solution to connect their buildings on the campus.

Customer complained that no-one told them that they could not use the cable outdoors and that we should take some of the responsibility and compensate them - But thankfully someone with a modicum of knowledge quickly shut that down on the customer side.

For me what was amazing is that it actually worked good enough for about 6 months !

We don't need maintenance this often, surely? Pull it. Oh dear, the system's down

Financial software - Money extraction

It's the mid 1990's and I have a customer using financial software for their payroll on a Unix system from a company that was selling their solution based on how cheap it was. The issue was not only was it cheap but also not very good. A major release was coming and my customer had invested heavily in getting the new release up and running and was literally banking on some of the new functionality. The new release is made available and the software vendor states that they will install for a small (Ah hem) premium, the price was astronomical and the customer said no - So the customer application team (1 guy) was tasked with the install and it did not go well, it seemed looking at what he had done to be very complex and he hit a wall on a strange error. We were asked to take a look, it was a very convoluted install and to be honest we could not make sense of it. So one of our coding guru's took a look and wrote a script to install the software, it worked ! It seemed to us the software company had a plan to extract a handsome premium for install services by having a stupidly complicated install instructions if customers attempted themselves. The software company was acquired by an outsourcing operation in the early 2000's, so I expect their legacy is no doubt thriving ..........

Typical '80s IT: Good idea leads to additional duties, without extra training or pay, and a nuked payroll system

Shadowing - What could go wrong

Working in the mid 1990's with a customer who was running a retail operation on amongst other things a VAX VMS Cluster (Best we do not mention their Ultrix DECsystem monstrosity and AS/400 bastion of how much can I spend with IBM). The VAX was two VAX 3000 series with DSSI disks in a RAID configuration 0+1, or in DEC words shadowed. The time had come to upgrade the VMS to the next major release and the plan we submitted was to have a plan (Which this customer was always adverse to as they thought this just increased the costs), take backups, work the upgrade, test and then release back to the customer.

But the customer had other ideas, that had a shadow, they could split it and hey presto a ready made backup .....- Customer :: Why don't you take down one half of the cluster, break the shadowing (RAID 0+1), upgrade and then switch over the systems, then bring the old VAX back up with a merge using the disk running the new version of VMS.

Now, as a technical challenge this is interesting, but has many opportunities for something to go wrong, either process based (i.e. getting commands wrong etc when breaking the shadow disks etc), but the customer would only take this approach.

Initially everything went well, the upgrade applied to the VAX not now in the cluster, old VMS VAX shutdown and then brought back into the new cluster running new version of VMS - There is a moment when you create a new shadow disk in VMS that was always a bit of a fingers crossed moment when it did the initial creation of the shadow pair, 99 times out of 100 no issues but of course this time an issue, there was some issue on the shadow disk creation and we had a fail, but also took down both disks, disks themselves OK, but some gremlin caused a software fail. The issue is that we only have the regular backup for restore, so all the changes in the system prior to upgrade where effectively lost and we had lost the upgrade and all associated effort.

We have covered ourselves with respect to the customer in that we had formally written that this was not an approach we recommended as we could not guarantee the data or upgrade - But as ever lots of noise from customer that this was our fault as it should have worked blah blah blah.

But lesson was learnt for 1-2 years that nothing beats have a project plan that also ensures you take backups before committing to a change and that RAID 0+1 (VMS Shadowing) is great but should not be used as a short cut for VMS upgrades in the future.

Are you sitting comfortably? Then we'll begin. Hang on, the PDP 11/70 has dropped offline

DataSolve - Sunbury?

Remember using their Mainframe bureau in Sunbury

So you really didn't touch the settings at all, huh? Well, this print-out from my secret backup says otherwise

It went wrong all by itself

Remember British Satellite Broadcasting (BSB) and their Squarerial (The square shaped satellite dish), a long since merged/defuncted UK corporation was using a VAX 11/785 in the development of this failed product (This is a whole story by itself in marketing pushing bizarre requirements to an engineering team) - The VAX 11/785 had been taken off contract as they had a lot of bright engineers and they would self maintain to save some money. So all seems quiet/good for about 8 months and we get the call, can we come to site the VAX 11/785 is borked and they need assistance (Which we will charge on a T&M basis). I arrive on site ask for some background as to what has happened, get basically a story that everything was good they had a problem but this problem quickly moved to a complete system failure. They had done some diagnostics but not touched the VAX physically.

The VAX 11/785 was switched off (Weird, not normally switched off, even when a fault is present) - Asked why it was switched off - They said they could not start the machine - The VAX 11/785 had a cutdown PDP that was the boot controller, so you would normally have this active even when the main machine had major issues.

So, my first action is to open the main cabinet door to look at the logic boards (The VAX 11/785 CPU was basically a set of logic boards connected to a backplane) - Opening the cover to the logic boards I asked again if they had done any "hands-on" troubleshooting - They said no, no-one had touched the machine at all.

So I said that was strange as one of the Logic Boards was inserted the wrong way round (Top was on the bottom rail) - This would have taken a lot of effort to do as the boards were keyed to go in easily one way only (i.e. with top of logic board aligned to the top).

Cue a lot of people suddenly looking guilty - They fessed up that they had taken the cards out to reseat them as they thought that may resolve the problem, they had a problem with one card being difficult to get back in, which in fairness was an issue with this enclosure - They thought the card was in the wrong way so flipped it an re-inserted and with what I can only assume was superhuman strength managed to get it all the way in.

It took some effort to prise the card out and peering at the backplane revealed crushed pins on the backplane.

So 2 days later, new backplane, assorted new logic boards and north of £35,000 they were back in business (let alone the business cost of not having the system up and running). They basically they had to spent about 8 years worth of the maintenance contract that they had cancelled in one go.

Their management did try to say that it was easy to make the mistake and it was our fault that this was possible (Negating what I can only assume was obvious visible guides not to do what they did, but the sheer brute force needed to do it). But sanity did prevail and they paid the bill and even put the system back on maintenance contract.

Alas th Squareial did not help the company make the expected millions (Especially as told to me by one of the lead engineers the spec they were instructed to build it to did not factor in cloud cover .... So in testing/demo phase all was good until it got cloudy - so expense re-design required) - And BSB got bought by Sky to become British Sky Broadcasting and the Squareial was consigned to the dustbin of "what were they thinking".

Hey, I wrote this neat little program for you guys called the IMAC User Notification Tool

Re: Usernames

Opportunity missed, it could have been first 4 letters of the first name followed by the surname :-)

When American Corporations Tried the Global Approach

Circa 1988 and the Wang Laboratories was rolling on globally their new hardware maintenance service they sold to their customers, I good friend of mine was trying extremely hard to get the name changed for the UK market as selling the WANG CARE service he thought might be difficult. He was of course ignored by the US based marketing team and all hell broke when released in the UK with a full advertising push etc.

Acronyms and Americans

Circa 2004 in Texas, there was an advanced training program - Technicians With Advanced Training - You could become a certified T.W.A.T - As the only Brit and seemly only person who spotted the issue (and kept very quiet), it was marvellous when this was rolled out beyond Texas ......

Crash, bang, wallop: What a power-down. But what hit the kill switch?

Beware the Chairman of the Bank

It's the early-mid 1990's we are an outsource supplier to a very large UK bank stock brokering operation - They have invested heavily in a clustered VAX infrastructure that has eliminated most of the single points of failure, except one big one in that the cluster (And shadowed disks) were in the same physical data centre and in fact in the same large bank customer only data center hall/room.

This clustered VAX infrastructure worked extremely well and the bank was very happy with the high availability, much improved speed and the service we were delivering, so much so, they wanted to bring a bunch of their senior executives to our facility to show them what the banks money had been spent on.

Cue to day of the visit, everything was going really well, the final part was an escorted visit to the actual data room where all their kit was housed - Some questions asked by executives but suddenly the chairman announced that he understood that it was impossible to shutdown the computers (Great selling of high availability by the account manager) and suddenly the chairman reached out to the big red button on the wall. Literally mill-seconds from disaster I managed to intercept the chairman's had (with quite a bit of force) and stop him hitting the EPO.

However the aftermath was me apologising for the "assault", the bank chairman thinking we had something to hide and overall 12 months to rebuild the customer confidence in us - Even though we did stop a complete powerdown which would have cost them hundreds of thousands.

Lesson learnt, never ever take customers into the data center even if they own it - So next step was a couple of windows installed so customers could see but not touch.

If I could turn back time, I'd tell you to keep that old Radarange at home

Re: Military Ship Radar

That is the one.

Military Ship Radar

Military Ship Radars are very powerful things, they should never be activated in port and being anywhere near the main array when active requires some serious lead underpants. Circa 1988 HMS Invincible is in Devonport Dockyard, the main radar array was accidentally activated for 2-4 full sweeps, at the time I worked for a Mini Computer company as a systems engineer, almost everything in line of sight that got a dose of the HMS Invincible radar sweep either crashed or had memory corruption. A very busy day, initially confused as to the cause of the issue, but as first system call was from Devonport Dockyard and the Radar Array activation was being talked about it didn't take long to connect the two events (And all the other calls coming in). I seem to recall there was some very serious disciplinary action taken on a number of the naval crew from HMS Invincible, it was a very serious matter and could have been worse for the crew and people nearby.