Posts by ColinPa

Bug that wiped customer data saved the day – and a contract

Accidental on purpose?

I was visiting a customer and was peripherally involved in a problem they had.

They had a home grown configuration file which had had change upon changed made to it - to the extent that no one knew how it worked. They had had meetings to discuss rewriting it, and thought the risk was too high of it not working.

I was at the customer for the weekend while they did a major upgrade, and they wanted all support people in. One person I was working with thought he could improve their configuration program.

At midnight, he found his changes made it worse; and then he found he was updating the master copy, and when he backed out his changes it didn't work. The previous backups were corrupted, so could not be used.

He bit the bullet and rewrote it from scratch.

I went to the hotel at 0600, and came back at 1500 for the next day's support. He was still there. In the middle of the night I found him asleep at his desk (for an hour or two).

Sunday night about 8pm the major upgrade was successful, and the rewrite worked.

The guy was hyperactive from coffee an doughnuts. He said that most of the work was understanding what was still needed, and what was junk left over from earlier systems.

His boss when told, was happy and annoyed (more happy than annoyed).

As I was checking some documentation on his laptop, I noticed some files saying backup of....

When I mentioned it, he said "you didn't see them I sort of caused the backups to be unreadable. I just wanted to fix this application"

Enforcing piracy policy earned helpdesk worker death threats

guns!

One colleague went to a conference in Dallas. He checked in, went to his room. He started to unpack, and opened the window to the outside world. A few minutes later a mirror in his room shattered. A "Drive by" had taken a pot shot at the hotel and into his room. He asked to be moved to a room on the top floor of the hotel.

A friend who worked in a "secure" company had an American colleague come to the UK on business - it was the American's first trip, so there were discussions like - yes, you need a passport.

One of the questions from the American, was "will there be somewhere to store my piece when in meetings?" After a few clarifying questions, my friend had to explain that you cannot bring your gun to the UK.

The American was persuaded not to fly to the UK, hire a car and drive 6 hours to the town where the meetings where. He might have been a lethal weapon. The American later admitted that NOT driving in the UK was a very good idea.

Tech support chap invented fake fix for non-problem and watched it spread across the office

Re: Hilarious!

Don't forget that red placebo pills are more effective than blue ones (or is it the other way around)

Techie banned from client site for outage he didn’t cause

Don't blame me - they virtually banned me

One of the service help desk came round to my desk, and said an important customers was blaming me on a major problem they had.

They had told their management, "we did xyz as expert abc said, and it caused this problem".

I looked into it.

The advice I had given was

If blah blah blah blah then do xyz. They ignored the if statement and did it anyway.

I had words with the salesperson who worked the account, who did some digging.

I offered a good will(free) call with the customer to discuss the problem and solutions.

The technical team had screwed up, and wanted to blame someone else, so they were definitely not keen on a call.

Help desk read irrelevant script, so techies found and fixed their own problem

ahh - that was embarassing

We had a big mainframe customer come to visit our lab, and discuss some problems they had with software that runs on many platforms.

One of the senior technical people, who develops on Windows, and had not used the mainframe said

Techie: "Change the config file, shut down your system down and restart it". That's what we tell our customers.

Customer: I dont think that will work, it will take about an hour to restart it

Techie: (very glibly) Well you should buy a faster box.

Customer: very, very, long pause

Customer: We have the fastest box. Our system come up in about 30 seconds. It takes an hour to check the connections to the 10,000 other system, and the 5000 database tables are all operational before we open it for business.

Techie : Ahhh. This was one of the situations where he wished he could just disappear.

Another long pause, before a senor managed stepped in to facilitate the discussion.

The techie was not involved with any future customer discussions

The Y2K bug delayed my honeymoon … by 17 years!

What goes up ... must come down

I had to fly to the US on Jan 2nd to support a very large customer in case they had any problems.

I heard people had been taken to hospital, because to celebrate the new century people fired their guns into the air.

What goes up... comes down, and there were a few cases of people being hit by the descending bullets

___________________

Over the century change, one problem we experienced was the coffee machines stopped work, so people working through the night could not get their dose of caffeine. The boss brought in his coffee maker.

Whitehall rejects £1.8B digital ID price tag – but won't say what it will cost

Re: I can do it for half that

70 million records is small. I did some work for a Chinese bank - they had over 200 million personal bank accounts. You just need to use the right systems ( and do not expect to use a spread sheet or similar technology).

The problem is the non functional requirements. If you write an record audit every time someone's record is accesses - that will need a database 100 times the size of the number of accounts, and you keep this information for 10 years. And you'll need a fail over and DR system....

And don't forget field encryption etc.

Spec?

Spec ... they have a spec ? Wow - can we all see it - or is it just chartware?

Linux admin hated downtime so much he schlepped a live UPS during office move

High availability

I worked on IBM's mainframe systems, and a customer came to present about their systems.

He said "we have provided an uninterpreted service to our customers, front end and back end, for over 3 years"

We gave a smattering of applause.

He said "Wait ... I haven't given you the good story. We've done this, upgrading all of the software twice, and moved machine room twice"

That got a good round if applause.

He explained they moved some machines from one machine room to another, brought them up. The front end routers then directed work to these systems. The database could be morphed over in a similar way.

We were well impressed

Developer battled to write his own documentation, but lost the boss fight

Sometimes getting it right is important

After being plagued by having people who could not spell, one guy tried the following. At the end of the day instead to paying someone £30 pound, he gave them £3 pound. When they complained he said "well it says £3.00" here". He argued with them - and then gave them their £30.00 From then on, people were much more careful.

The usability seesaw

I worked on a large, very popular project. The installations documentation was 30 pages of instructions, then try to start it. It would often barf "ERROR IN $$BOPEN" and not tell you the file name.

For people who had never user it before this was very bad, as they had to start all over again. There were customer reported problems, which the pubs people tried to fix.

Experienced people would not have these problem because they just copied what they did last time, and changed a few names.

I rewrote this in baby steps. Do the following things, check it works. Do the next thing, check it works Do the next thing check it works - it doesn't? - well check the last thing you did.

A developer took this and improved it and we gave it to the pubs department, who made some changes and shipped it.

The response from customers was positive and the number of problems reported on installation went down significantly.

I changed projects. 5 years later I was asked to help a customer and found that the documentation had been "improved" by the new function owner who took out all of the baby steps and so the documentation was now 60 pages of instructions followed by "now try starting it".

I spoke to the new owner who said he thought that all the baby steps were unnecessary. He tested it every release - with the definitions he used the previous time. He had never started from scratch and type every thing in. And yes - there were lots of problems reported by customers.

It felt like a seesaw. Either make instructions clearer, but longer, or make then shorter and less clear.

The person who writes documentation needs to be empathatic - being able to put themselves in the position of someone who knows nothing.

Networking students need an explanation of the internet that can fit in their heads

Give the requirements

As well as the first statement, I would cover what people expect today.

Bits of the network will need to be replaced, for example your home router is connected to your service supplier. If they change their hardware, it should be transparent to you The "network" should carry on working.

You should not be impacted by other people. If the people next door are hammering the network, it should not affect you. If there is a lack of capacity they should be slowed down - not you.

When I stream my latest show - I expect it will display without hangs.

If you pay for a premium service, it should continue to work even if someone cuts an undersea cable.

Techie ran up $40,000 bill trying to download a driver

Dont hone home

I remember the days before WIFI when you had to connect your laptop to the phone cable. We had an AT&T dialer so we could phone a local number to get back to base.

It was someone's first trip abroad, to present a conference in a big hotel in the US.

Half an hour before his session he phoned home, got the demo working ... did his presentation working and did his demo. Round of applause - hurrah

His phone bill was bigger than than his travel. He had not used the dialer - so was charged at International rates for 90 minutes, with a hotel marlup of + 50 %

Frustrated consultant 'went full Hulk' and started smashing hardware

Delete is a two stage process

I learned early on that if you need to delete something. Rename it or move it. Then a week or so later delete it.

I learned this the hard way deleting a file. We easily recreated the file but is was missing some special attributes we hadn't noticed, and so it didnt work.

Great manager

I had spent a week at a banking customer abroad, and had a free day at the end before I flew home.

My phone rang at 0700 on Saturday, and my "boss" asked if I could help with a problem to which I said yes, and he gave me the contact number of the technical team. He said if I get unwanted calls phone him. If I need >anything< phone him.

I worked with the customer's teams agreed the problem and the plan to fix it. I told our boss what the plan was, and updated him as we completed each step.

I phoned him "we need access to ..." 2 minutes later we had access. "We needs someone to..." done.

All systems were back by 10am. I found that the bank had been down since 0400. The outage had the attention of senior managers (and the board). They had a plan for when this happened. One person would act as the focal point getting all the phone calls from people who cannot help. Only this person can phone the technical team.

It worked a treat - and let the technical folks handle the problem without the distracting chaff.

A great manager to work for. It was good we did not know the scale of the problem... we just got on and fixed the problem(s).

As we executed the plan my boss could tell interested parties the progress we were making, so everyone felt it was under control.

UK rethinks offshoring ban for £8M online procurement system

What does keeping this data in the UK mean

Does this mean

- all servers, are in the UK

- all backup systems are in the UK

- all data backups are in the UK

- all people who look at data - so any support people - are in the UK... so the data cannot be seen from India or Europe of America?

I worked in an international company, and we had 24* 7 support from people around the world.... I guess this will not be allowed.

The networking experts were in the US... they are not allowed to see the data.

Techie found an error message so rude the CEO of IBM apologized for it

Impossible code

I remember someone who had some code like

if bit0 = 0 then ...

else if

bit0 = 1 then ....

else

Phone George on 0196281....

10 years later George got a phone call. A developer had carelessly changed the field from a one bit field to a 2 bit (or integer) field and so his else clause was fired.

UK's digital hospital plan meets analog reality check

"We want to be assured that no one will be left behind in this new online hospital," he said.

Don't hamstring them from day 1. Getting 10% using it - and finding the problems would be a major start.

It is not that the current system works. My aunt had to take two buses to get to and from a specialist hospital and it took all day.

After a week of this, my brother took time off work to take her ... and that was 3 hours round trip.

He would have been happier to take a laptop to my aunt, set it up for her, and leave her too it.

Word to the wise: Don't tell your IT manager they're not in Excel

Some managers are OK,

I was a new hire 40 years ago, and had been asked to copy some documents and do it duplex, so there was writing on both sides.

I printed the first page (10 times), then took the paper and put the paper back into the photocopier, and printed the next 10 pages. It was easy once you worked out which way the paper had to inserted.

My manager came to do some copying, and having watched me struggle - asked "Why don't you use the duplex button" ... which does this automatically. I remember this 40+ years later.

A long time later, when I had a not very good manager, he wanted some stuff copied urgently. I passed him at the copier, and he was saying "damn machine has a blockage... I'll go and find another copier".

I suggested it might be easier to unblock it, but he didn't know how. I opened the doors and written inside was "if this light is on - then pull lever 1, clear the blockage and then push level 1 back".

We followed the easy instructions and cleared the blockage. My manager was not keen on seeing the sheet which caused the blockage - but I could tell it was not work related.

As a so called "technical manager" he was not very technical - he thought we was very technical, and hated me showing him up.

Techie ended vendor/client blame game by treating managers like toddlers

Ping pointed the finger

I was involved in a set of long standing problems with a customer. We were a networking product, and the customer complained that periodically the throughput dropped almost to zero - and by the time they came to look at the problem it got better.

I think the customer found our support teams friendlier than their own support team.

I got them to have a simple program which every minute did a ping to the the remote end, and capture the response time.

They sent me the output, and I could see normally good response times with the occasional spike.

Our emails crossed mine said "did you have a problem at 0713" theirs said "we had another occurrence at 0713".

The customer asked me to present to their network support team, and it took about half an hour.

After we had finished we got push back from them "we do not see any errors/problems in our logs"

me: "What response times were you capturing for this time period"

tap tap tap.. "ahh we are just going on mute"

Eventually they said "we don't have any monitoring enabled on that part of the network" - which is why no errors were reported!

The networking people found the problem and the throughput problem was resolved.

The people I was working with still kept the ping every minute.

I was a part-time DBA. After this failover foul-up, they hired a full-time DBA

You should do this

I remember getting into trouble when the documentation said "you should do .... " when it meant "you must do...". The word 'should' implies recommendation but not mandatory.

Fortunately it was only a test system we could easily recreate. My supervisor got the documentation changed.

CIO made a dangerous mistake and ordered his security team to implement it

You are very brave

I had to visit a very large US customer about a problem on the mainframe. The team were depressed because the new big boss had told them to move off the mainframe on to commodity hardware. Commodity hardware means if you need more capacity you go down to your local Walmart and buy it. The big boss said it would save big bucks once the transition was done.

As part of my problem resolution I had to report to the upper management chain. I also spoke about my views on their change to commodity hardware.

I made comments like "you are very brave" (as in Yes Minister), and "I do not think you understand the capacity you need".

Someone kindly took me aside and explained how their estimates were right. They looked at the amount of traffic into the mainframe on the busiest day, and used that as their base line.

My comments like "on the mainframe 90% of the traffic between applications and the data base are cross memory and is not measured" fell on deaf ears. (Mouth engaged, brain in neutral, ears turned off)

They got a taste of things to come when they ran tests with live data and found the network could not cope, fixing the network then showed they didn't have enough commodity hardware.

Overall the project cost more than the mainframe - but they could not go back a) the mainframe people had all left, b) it would have embarrassed the people who wanted the new scheme. c) they would have all this kit they didn't need.

I used to meet up with someone at a conference, and hear all the news.

Basic projector repair job turns into armed encounter at secret bunker

How did you get in here?

This was a trip to a banks HQ - we were meant to be escorted at all times.

Our last day (after a couple of week visit) we could see the exterior badge locked door (for staff) was green - so we went in. We only had paper badges saying visitor.

We went to the secure inner sanctum, and the door was open because the cleaner needed the plug outside the room - to be able to clean inside the room.

We sat down and waited. Our contact came in and asked - "how did you get in here?" so we told him.

It turned out there had been a power blip somewhere - and the doors failed safe to unlocked.

They installed a "plug for cleaners" inside the inner sanctum.

A colleague in the US went to a secure underground nuclear bunker site for a briefing. He was accompanied everywhere at all times by an armed marine. When I say everywhere, the toilet cubicles did not have doors on them!

He said the whole experience was very scary.

I was once briefed by a man on security, who did not exist..... he said he worked for our company, but was not in the internal phone directory.

Why the UK public sector still creaks along on COBOL

Understand the problem - a little knowledge is a dangerous thing

I went to visit a US government establishment, to discuss some scenarios about the mainframe. They were not happy because the new boss person had been tasked with reducing the IT costs by 50% or more, and knew it could be done because he had set up a web server on his home Windows machine. He saw the solution was "to spin up utility hardware" (that is PCs you get from a PC store) when there was a demand for more capacity.

The IT guys I worked with did their own sums and said the amount of kit they would need would cost more than the existing mainframe - so no savings.

With the mainframe they had a down time of about 5 seconds of unavailability a year, processed 1000s of transactions a second and the response time was sub-second. They a backup site, and would regularly switch to it.

6 months later I got involved with a performance problem with the new environment. There were intermittent 10 seconds hangs many times a day and I could see it was a disk speed problem (in effect I had a trace of every I/O and about 10% had hundreds of milliseconds response time). The people who managed the virtualisation and disk farm said their stats were good.

I heard they solved their problems by doubling the size of disk farm, tripling the network capacity (to handle all the machine to machine communications), and disabling logging. At a huge $ cost.

I also know of another customer who moved from COBOL on CICS to Java on CICS (on the mainframe) for some parts of their business. They did a lot of planning and prep work - and the work came in on time - and budget. They started with a small, not critical piece, of code - and got that working - and learned many lessons from it. They then did a bigger project, applying the lessons learned etc, by the time they came to the critical code, they had a well established process.

It can be done.

Joke: How do you eat an elephant? A mouthful at a time. .... that applies to IT projects.

Can someone remind me ...

Can someone remind me ... Is it replace old systems with known bugs - with new systems with unknown bugs. Move fast and break things?

People often forget that the test suite is often larger than the "product" it tests. It has to do all the edge cases, performance, scalability, security, security validations (no you cannot have access) reporting, encryption of fields etc. The functions that "no one uses". A bank turned off a server which was "never used". It was used once a year at audit time for about 4 days.

If you go from screen scraping enter customer number in the first field of the old system, to its now the third field in the new system, this soon becomes a problem.

Dividing a system so you can isolate components, and have clean interfaces between components can make some of these transformations easier. Start by converting green screens to http and web pages ,so you can have named fields. This makes automation much easier.

Someone said that some of these projects are like trying to replace the engine of a jet - whilst in flight.

Microsoft pushes Pull print, so you don't have to dash to the printer to grab the 'Fire everyone' memo

What's new?

10 = years ago we had a Linux solution where you had to place your badge on the printer to authorise the printing of your output.

Pay attention, class: Today you’ll learn the wrong way to turn things off

Whoops goes the EPO

Someone told me about a major outage caused by moving some disks. This was 30 years ago when disks were boxes 6 ft high and 3 ft square.

There was not quite enough space in the machine room for some new kit - it was a matter of inches too small.

No problem. Two of them put their backs against the problem disk, and having been on the lifting course, pushed with their legs.

It worked!

The problem was that one of the guys got his belt caught in the Emergency Power Off button (which you have to pull to activate).

As they stood up there was a sudden silence around them.

It took a good few hours to recover.

They had been proud of their system availability, and had charts on the wall. Their carelessness was on display for a year, until it rolled off.

Servers hated Mondays until techie quit quaffing coffee in their company

You are a director ? Oh yes - there is a problem

We had a nice office with a great view, but the only problem was this was over an outside corridor. When it was cold outside we got cold feet, so we brought in insulation, and put a carpet on top. ( We did wonder why lowly minions were given such a nice office - every one else knew about the office). We complained to site services who said - there was not a problem - and we were wusses.

A director took over our office (and the ones next to it). The first cold day she complained the floor was cold - and whoosh site services were round "oh yes there is a problem". They found that water from the rain pipe from the roof was leaking into the space under the floor. When it was cold - it froze - and the cold flowed up to the floor.

It took them a few days to fix the leak - and a few months to replace all the insulation along the corridor - having to work from the outside.

Problem PC had graybeards stumped until trainee rummaged through trash

Why are books organised by height or colour?

If you look at the fashion magazines showing how to furnish the room. The book shelves are always ordered, by height, or colour. (or an absence of books) These people clearly do not read their books.

Most people arrange them by author. I've got a row of books for an author by one publisher, plus another book in the series by a different publisher, which is a different size and colour scheme.

The tiny tech tribe who could change the world tomorrow but won't

Dyslexia rules not ko

I remember seeing a site talking about which fonts and formatting to use that make it easy for people with Dyslexia to read.

The site didn't use the fonts..but the standard ones which are hard to read!

Write-back to aging UK health systems lessens benefits of Palantir-based platform

Whoops we forgot to state all the requirements up front - again

I remember "lessons learned" from several major projects which failed spectacularly. On of the common lessons was "understand and document the requirements before talking to vendors"

The requirements should be like "we want to ..." rather than "use this tool to..."

It sounds like the write back requirement was not documented.

Yes, I wrote a very expensive bug. In my defense I was only seven years old at the time

Don't send data if no one is listening.

My company found that the external site that sent most data to our site was the local football team which was sending MB/second!

When you are looking at a site, it can send you data, such as a 5 seconds clip showing a goal being scored, or a ticker tape.

Normally the remote site only sends data when the web page is active.

Their web page did not have this, so if the web page was hidden by other pages, it still kept sending the 5 second clip of the goal being stored.

Many (hundreds) of people would check the web page first thing, while drinking their first coffee of the day, and leave the web page there.

This was pointed out to the foot ball team, and they fixed it - and were very grateful , because their traffic dropped by a factor of more than 1000, and they saved lots of money

'Trained monkey' from tech support saved know-it-all manager's mistake with a single keypress

"I'm not stupid you know"

One of my bosses was in tech sales before he became a manager. I worked on the mainframe down in the bowels with the really technical stuff.

With my managers I used to say "there is a problem with customer ABC it is really technical... so I wont explain it" - and most managers were happy.

This manager said "I was in tech sales - so I'm not stupid, please explain the problem".

So I did - there is a timing problem between this widget, and that widget resulting in.... I knew within seconds that he was totally lost, but he kept nodding.

After I finished he said immediately "I'd also like to talk to you about..."

I heard he asked one of my junior colleagues about the problem, and got the answer "you don't want to know ... it is deep shit that no one round here understands".

He continued to think I was deliberately making it complex to make him look stupid.

UK charity bank CAF branded a 'disaster' after platform migration goes wrong

"after years of planning, preparing, and testing"

It could be that they didn't actually know how the customers used the product.

With one product I was involved in, you could set a time out value tenths of a second. One customer used our product because the standard software only had a time out value in seconds!

They didn't use 80% of our product.

I remember going to customers and thinking "you are doing what! with it" - and going back to tell the test teams.

Junior sysadmin’s first lines of code set off alarms. His next lot crashed the company

and that's why they moved me to where I could do less harm

When I first worked nearly 50 years ago disk space was expensive, and I worked on DOS on the mainframe.

I was in the build group compiling a product. We had the build system and the live system.

I followed the instructions, deleted the old product, and started compiling the new code.

10 minutes later someone wandered round saying there was a problem.

I was working on the live system and not the build system.

My supervisor sorted out the problem and every one was happy.

They changed the instructions to say change this line.... to point to the build system.

Next week, I duly followed the new instructions and 10 minutes later someone came round...

I was building into the live system again.

My supervisor sorted it out - and changed the instructions to say "change >both< occurrences of..."

and that's one reason why I was moved to another platform with read only disks.

They also rewrote the instructions to include "what everyone knows".

Techie went home rather than fix mistake that caused a massive meltdown

metric/imperial

My father went to buy some timber.

Him:"I'd like a plank 6 ft long"

Them:"We are metric"

Him:" Ok I'll have a 2 meter plank"

Them:"We dont sell that size. We can let you have 1.8 meter plank" ( = 6ft)

Re: Honestly I'm bilingual

The moon may not have been visible at that time.

Re: Honestly I'm bilingual

We also have collaboration between US and Europe. I think there was one space shuttle mission looking for a mountain 4200 m high. So instead of looking for Hawaii (4200 meters high) - it turned towards the sun .. the closes thing to 4200 mile high.

Re: Honestly I'm bilingual

Ladies and gentlemen - this is your captain speaking. We will be flying at 40,000 ft at 1000 km per hour.

It's hot today - it is in the 20s...do you remember it being in the 80s when you were young.

Economists sceptical over UK Spending Review's partly AI-driven 10% budget cuts

I don't want plans I want results

I was involved (on the perifery) of a project to save money on a project. It was announced to everyone so we were all "part of it". Teams to produce plans.

The manager I was working with, called a meeting, and said "I don't want plans... I want actuals". We met up again a week later and had results like

1) because I always had to approve xyz - Ive set up a script to logon once a day and do it

2) we have all these old copies of software on disk - which we think no one wants. We've archived them to tape and freed up xGB of disk space

3) We don't think anyone uses these servers. We've powered them off. We can easily power them on again if anyone complains. We can virtualise them if needed.

All small stuff - but the first steps are the hardest.

A month after the big announcement the senior manager calling for the project to save money, asked for a review on progress so far.

Departments put up nicely written presentations about what they could do. etc

My manager put up one chart with the above points and said ( verbatim ish - it took less than 2 minutes)

- these are actuals. We are not doing any planning

- in any status meeting - people are allowed at most 1 chart

- the meetings now run to time

- we all go for coffee at 1000 - and raise any issues then - saving half an hour a day * 10 people

The senior manager took the hint and realised how much time (and cost) had been wasted doing blue sky thinking, and preparing charts etc.

He gave some ground rules

- No more overtime. Schedule your work better - or ask for help.

- I want to hear about meetings which are a waste of time

- Simple charts

- Get to know people in other areas who may be able to help you ( no more silos)

- I will be walking around the building - if you see me - tell me something (good or bad)

As a result meetings shrank (they had meetings but with 3 people not 13) and they found lots of overlap and so could save duplication.

Simple stuff but it worked.

Techie traveled 4 hours to fix software that worked perfectly until a new hire used it

Insanity

"the definition of insanity is doing the same thing again and again and expecting a different outcome"

Sometimes I do something every day, and it works. Every now and again - it does something different.

And the opposite. Something is broken, so I start to debug it - and now it works.

Timing problems are a curse.

Man over board

I did a day skipper course for how to manage a yacht. One exercise is "man overboard" which is a bucket, a float, and stick with a flag on the top. You learn how to turn the boat round, and get back to the MOB and pick it up. We all passed.

At lunch, we were anchored, having a nice lunch, I asked if we were to be taught how to get someone out of the water. No -- that's not in the syllabus.

So being in the med, in the middle of summer, I jumped over board saying "Man over board".

Every one burst into action

They threw me a rope. my hands were so cold I could not grip it.

Someone tied a few knots in a rope and threw it - again I could not grip it

Someone tied loop in the rope and threw it to me. Great - I put it under my shoulders and they tried heaving me out - they could not.

25 minutes later someone had the idea of using the mainsail winch - and I got out.

In real life I would have died.

If you have procedures you have to test them!

I was at a customer and some emergency stuff (key to the power room?) was locked in the key press. During practice it all worked fine. When they had a real emergency, the shift manager with the key was in a meeting with the senior managers. The senior managers were surprised when three guys burst in unannounced saying "we need the key - NOW" - and every one ( including the senior managers) rushed back to the ops room to help restart the systems.

Another customer had a wonderful document about what to do in an emergency - it was well tested, and online. After a power outage, they could not access this document because the server it was on was not active. Fortunately some one had a old paper copy which said ... start these server first... then you can access the document!

I touch it and it breaks!

As a tester I got a reputation for breaking things. I was always impatient, and would multi task ( use two windows) when one of the tools was slow.

We had to use a new tool for entering what you had achieved over the year. Because there was no save button, I assumed, like Gmail, it had autosave. I submitted it to my boss who said it was empty.

I did it again...and again it was empty.

I contacted the help desk, who asked "did you press the save button?" me:"I don't have a save button"... "Did you press the help button ?" "I dont have a help button"

If you run the application full screen, all of the buttons appeared, if you run it in a narrow window, all of the buttons disappear.

With all of their testing they naturally used full screen, so naturally didn't find any problems.

I had a call with the development team, and mentioned the 50 "defects" I found. Such as

me:"there is an icon with 3 small circles - what does it mean? There is no hover text, and it is not mentioned in the help"

them: "It is meant to be obvious...",

me "well it has failed - so what does it mean".

them: "err we don't know"

me:"The fonts you use are not dyslexia friendly, and the fonts are too small and not changeable"

them "Err"

me: "Grey text on a grey background is not easy to read in a well lit area " ....

UK Spending Review prescribes £10B digital remedy for NHS

Re: Waste

It is easy to start afresh. The hard part is migrating existing data to the new systems, bearing in mind it may be in different formats. Big endian -little endian, dates - images.

You might spend as much writing the conversion routines.

Some of the Nordic countries have very good systems. Can we buy/rent from these people.

UK govt promises digital reform in spending review. We've heard that before

measurement is only a step

Measuring may have no effective effect.

I was on a help desk. We had a slow-close policy. We would leave a problem open for a week, and close it if the customer did not call back.

We were told the number of open calls was too high - we needed to fix it.

So we closed the calls once the problem was resolved. The number of open calls dropped. If the solution didn't work, the customer now opened a new call. So now there were more calls.

We were told to reduce the total number of calls. So we reimplemented the slow-close policy, etc.

The solution was not the measurement of calls, but to fix the documentation/quality of the underlying code!

Techies thought outside the box. Then the boss decided to take the box away

A room with a view

In the days when we had offices, there were some offices with wonderful views.... and our team was allocated them. We enjoyed the offices until winter, when we got cold feet because our offices were over an exterior corridor, and so the cold came up though our floor.

We reported it and were told cold feet is the price of a nice view. (We now understood why we were given the offices). Still we managed, we brought in carpets etc and were happy bunnies.

Until one Christmas when a senior manager wanted our offices merged into one big one.

Two weeks after our move out, the manager obviously reported the cold floor problem as there was suddenly a lot of investigation of the floor.

It turned out that there was a (drainage? ) pipe which was full of water which froze when it was cold... hence the cold floor!

We thought it was interesting that we reported it and were ignored. The seniors manager reports it - it is investigated next day.

User unboxed a PC so badly it 'broke' and only a nail file could fix it

Don't press the enter key

I was at customer supporting a mid range system (it would support about 100 end users).

The boot sequence was usually

- press the IPL button on the CPU

- swing you chair round, and press enter on the operator console. The first display to hit enter was the operator console

- it would prompt "enter IPL parameters"

Except one day when I was there.

- press the IPL button on the CPU

- swing you chair round, and press enter on the operator console - nothing happened

we repeated it, and had the same result.

A few minutes a guy from the help desk wandered in saying "Ive got a guy on the phone who says his screen keeps saying "enter IPL parameter"

The guy was sitting at his desk, bored, so sat there with his finger on the enter key.

He was educated to only touch his keyboard when the logon screen came up!

Ex-NSA bad-guy hunter listened to Scattered Spider's fake help-desk calls: 'Those guys are good'

Dont you know who I am?

My father was in the Royal Navy on ashore based ship. It was a ratings first day on guard duty.

One day this guy arrived at the gate, and the conversation went

Him: let me in, Ive left my badge in my office

Guard: No sir - no badge - no entry

Him: But I'm the captain of this ship

Guard: Sorry sir - I cant do it

Him: I'm the captain of this ship - dont you know me

Guard: Sorry - no sir -its my first day

Him: What's your name... .. now call the officer of the watch

OOW: Ah Hello Captain - what's the problem?

Next day the rating was summonsed to the captain's office. This usually meant punishment!

Captain: Well done for not letting me in. I left my badge behind, but if you had let me in, you would be here on a charge, rather than a commendation. Well done.

Automatic UK-to-US English converter produced amazing mistakes by the vanload

Whoops

On my first trip to the US I had to present to a room full of developers. I had given it several times in the UK. After an hour I put up a chart saying "Fag break" - every one sat there bemused.

I said "Cigarette break?!" and every one got up.

I also learned on that trip that you park on the driveway and drive on the parkway.

At a conference where there was simultaneous translation from English into French, German and Italian, someone was going though a dump

"At offset Baker Dog Dog"... there was laughter as the French heard ".. Boulanger, Chien, Chien"

Dilettante dev wrote rubbish, left no logs, and had no idea why his app wasn't working

Covid

Having read all of the comments, it made me think about the software used to guestimate the spread of Covid. It had once worked... and then been added to multiple times, by many people, and after it was "improved for publication" it was still hard to follow.

They had problems like - run it twice with the same input, it gave different results. So there was clearly some problems ( because if you run it twice you should get identical results).