Posts by ColinPa 585 publicly visible posts • joined 23 Jul 2018
After being plagued by having people who could not spell, one guy tried the following. At the end of the day instead to paying someone £30 pound, he gave them £3 pound. When they complained he said "well it says £3.00" here". He argued with them - and then gave them their £30.00 From then on, people were much more careful.
I worked on a large, very popular project. The installations documentation was 30 pages of instructions, then try to start it. It would often barf "ERROR IN $$BOPEN" and not tell you the file name.
For people who had never user it before this was very bad, as they had to start all over again. There were customer reported problems, which the pubs people tried to fix.
Experienced people would not have these problem because they just copied what they did last time, and changed a few names.
I rewrote this in baby steps. Do the following things, check it works. Do the next thing, check it works Do the next thing check it works - it doesn't? - well check the last thing you did.
A developer took this and improved it and we gave it to the pubs department, who made some changes and shipped it.
The response from customers was positive and the number of problems reported on installation went down significantly.
I changed projects. 5 years later I was asked to help a customer and found that the documentation had been "improved" by the new function owner who took out all of the baby steps and so the documentation was now 60 pages of instructions followed by "now try starting it".
I spoke to the new owner who said he thought that all the baby steps were unnecessary. He tested it every release - with the definitions he used the previous time. He had never started from scratch and type every thing in. And yes - there were lots of problems reported by customers.
It felt like a seesaw. Either make instructions clearer, but longer, or make then shorter and less clear.
The person who writes documentation needs to be empathatic - being able to put themselves in the position of someone who knows nothing.
As well as the first statement, I would cover what people expect today.
Bits of the network will need to be replaced, for example your home router is connected to your service supplier. If they change their hardware, it should be transparent to you The "network" should carry on working.
You should not be impacted by other people. If the people next door are hammering the network, it should not affect you. If there is a lack of capacity they should be slowed down - not you.
When I stream my latest show - I expect it will display without hangs.
If you pay for a premium service, it should continue to work even if someone cuts an undersea cable.
I remember the days before WIFI when you had to connect your laptop to the phone cable. We had an AT&T dialer so we could phone a local number to get back to base.
It was someone's first trip abroad, to present a conference in a big hotel in the US.
Half an hour before his session he phoned home, got the demo working ... did his presentation working and did his demo. Round of applause - hurrah
His phone bill was bigger than than his travel. He had not used the dialer - so was charged at International rates for 90 minutes, with a hotel marlup of + 50 %
I learned early on that if you need to delete something. Rename it or move it. Then a week or so later delete it.
I learned this the hard way deleting a file. We easily recreated the file but is was missing some special attributes we hadn't noticed, and so it didnt work.
I had spent a week at a banking customer abroad, and had a free day at the end before I flew home.
My phone rang at 0700 on Saturday, and my "boss" asked if I could help with a problem to which I said yes, and he gave me the contact number of the technical team. He said if I get unwanted calls phone him. If I need >anything< phone him.
I worked with the customer's teams agreed the problem and the plan to fix it. I told our boss what the plan was, and updated him as we completed each step.
I phoned him "we need access to ..." 2 minutes later we had access. "We needs someone to..." done.
All systems were back by 10am. I found that the bank had been down since 0400. The outage had the attention of senior managers (and the board). They had a plan for when this happened. One person would act as the focal point getting all the phone calls from people who cannot help. Only this person can phone the technical team.
It worked a treat - and let the technical folks handle the problem without the distracting chaff.
A great manager to work for. It was good we did not know the scale of the problem... we just got on and fixed the problem(s).
As we executed the plan my boss could tell interested parties the progress we were making, so everyone felt it was under control.
Does this mean
- all servers, are in the UK
- all backup systems are in the UK
- all data backups are in the UK
- all people who look at data - so any support people - are in the UK... so the data cannot be seen from India or Europe of America?
I worked in an international company, and we had 24* 7 support from people around the world.... I guess this will not be allowed.
The networking experts were in the US... they are not allowed to see the data.
I remember someone who had some code like
if bit0 = 0 then ...
else if
bit0 = 1 then ....
else
Phone George on 0196281....
10 years later George got a phone call. A developer had carelessly changed the field from a one bit field to a 2 bit (or integer) field and so his else clause was fired.
Don't hamstring them from day 1. Getting 10% using it - and finding the problems would be a major start.
It is not that the current system works. My aunt had to take two buses to get to and from a specialist hospital and it took all day.
After a week of this, my brother took time off work to take her ... and that was 3 hours round trip.
He would have been happier to take a laptop to my aunt, set it up for her, and leave her too it.
I was a new hire 40 years ago, and had been asked to copy some documents and do it duplex, so there was writing on both sides.
I printed the first page (10 times), then took the paper and put the paper back into the photocopier, and printed the next 10 pages. It was easy once you worked out which way the paper had to inserted.
My manager came to do some copying, and having watched me struggle - asked "Why don't you use the duplex button" ... which does this automatically. I remember this 40+ years later.
A long time later, when I had a not very good manager, he wanted some stuff copied urgently. I passed him at the copier, and he was saying "damn machine has a blockage... I'll go and find another copier".
I suggested it might be easier to unblock it, but he didn't know how. I opened the doors and written inside was "if this light is on - then pull lever 1, clear the blockage and then push level 1 back".
We followed the easy instructions and cleared the blockage. My manager was not keen on seeing the sheet which caused the blockage - but I could tell it was not work related.
As a so called "technical manager" he was not very technical - he thought we was very technical, and hated me showing him up.
I was involved in a set of long standing problems with a customer. We were a networking product, and the customer complained that periodically the throughput dropped almost to zero - and by the time they came to look at the problem it got better.
I think the customer found our support teams friendlier than their own support team.
I got them to have a simple program which every minute did a ping to the the remote end, and capture the response time.
They sent me the output, and I could see normally good response times with the occasional spike.
Our emails crossed mine said "did you have a problem at 0713" theirs said "we had another occurrence at 0713".
The customer asked me to present to their network support team, and it took about half an hour.
After we had finished we got push back from them "we do not see any errors/problems in our logs"
me: "What response times were you capturing for this time period"
tap tap tap.. "ahh we are just going on mute"
Eventually they said "we don't have any monitoring enabled on that part of the network" - which is why no errors were reported!
The networking people found the problem and the throughput problem was resolved.
The people I was working with still kept the ping every minute.
I remember getting into trouble when the documentation said "you should do .... " when it meant "you must do...". The word 'should' implies recommendation but not mandatory.
Fortunately it was only a test system we could easily recreate. My supervisor got the documentation changed.
I had to visit a very large US customer about a problem on the mainframe. The team were depressed because the new big boss had told them to move off the mainframe on to commodity hardware. Commodity hardware means if you need more capacity you go down to your local Walmart and buy it. The big boss said it would save big bucks once the transition was done.
As part of my problem resolution I had to report to the upper management chain. I also spoke about my views on their change to commodity hardware.
I made comments like "you are very brave" (as in Yes Minister), and "I do not think you understand the capacity you need".
Someone kindly took me aside and explained how their estimates were right. They looked at the amount of traffic into the mainframe on the busiest day, and used that as their base line.
My comments like "on the mainframe 90% of the traffic between applications and the data base are cross memory and is not measured" fell on deaf ears. (Mouth engaged, brain in neutral, ears turned off)
They got a taste of things to come when they ran tests with live data and found the network could not cope, fixing the network then showed they didn't have enough commodity hardware.
Overall the project cost more than the mainframe - but they could not go back a) the mainframe people had all left, b) it would have embarrassed the people who wanted the new scheme. c) they would have all this kit they didn't need.
I used to meet up with someone at a conference, and hear all the news.
This was a trip to a banks HQ - we were meant to be escorted at all times.
Our last day (after a couple of week visit) we could see the exterior badge locked door (for staff) was green - so we went in. We only had paper badges saying visitor.
We went to the secure inner sanctum, and the door was open because the cleaner needed the plug outside the room - to be able to clean inside the room.
We sat down and waited. Our contact came in and asked - "how did you get in here?" so we told him.
It turned out there had been a power blip somewhere - and the doors failed safe to unlocked.
They installed a "plug for cleaners" inside the inner sanctum.
A colleague in the US went to a secure underground nuclear bunker site for a briefing. He was accompanied everywhere at all times by an armed marine. When I say everywhere, the toilet cubicles did not have doors on them!
He said the whole experience was very scary.
I was once briefed by a man on security, who did not exist..... he said he worked for our company, but was not in the internal phone directory.
I went to visit a US government establishment, to discuss some scenarios about the mainframe. They were not happy because the new boss person had been tasked with reducing the IT costs by 50% or more, and knew it could be done because he had set up a web server on his home Windows machine. He saw the solution was "to spin up utility hardware" (that is PCs you get from a PC store) when there was a demand for more capacity.
The IT guys I worked with did their own sums and said the amount of kit they would need would cost more than the existing mainframe - so no savings.
With the mainframe they had a down time of about 5 seconds of unavailability a year, processed 1000s of transactions a second and the response time was sub-second. They a backup site, and would regularly switch to it.
6 months later I got involved with a performance problem with the new environment. There were intermittent 10 seconds hangs many times a day and I could see it was a disk speed problem (in effect I had a trace of every I/O and about 10% had hundreds of milliseconds response time). The people who managed the virtualisation and disk farm said their stats were good.
I heard they solved their problems by doubling the size of disk farm, tripling the network capacity (to handle all the machine to machine communications), and disabling logging. At a huge $ cost.
I also know of another customer who moved from COBOL on CICS to Java on CICS (on the mainframe) for some parts of their business. They did a lot of planning and prep work - and the work came in on time - and budget. They started with a small, not critical piece, of code - and got that working - and learned many lessons from it. They then did a bigger project, applying the lessons learned etc, by the time they came to the critical code, they had a well established process.
It can be done.
Joke: How do you eat an elephant? A mouthful at a time. .... that applies to IT projects.
Can someone remind me ... Is it replace old systems with known bugs - with new systems with unknown bugs. Move fast and break things?
People often forget that the test suite is often larger than the "product" it tests. It has to do all the edge cases, performance, scalability, security, security validations (no you cannot have access) reporting, encryption of fields etc. The functions that "no one uses". A bank turned off a server which was "never used". It was used once a year at audit time for about 4 days.
If you go from screen scraping enter customer number in the first field of the old system, to its now the third field in the new system, this soon becomes a problem.
Dividing a system so you can isolate components, and have clean interfaces between components can make some of these transformations easier. Start by converting green screens to http and web pages ,so you can have named fields. This makes automation much easier.
Someone said that some of these projects are like trying to replace the engine of a jet - whilst in flight.
Someone told me about a major outage caused by moving some disks. This was 30 years ago when disks were boxes 6 ft high and 3 ft square.
There was not quite enough space in the machine room for some new kit - it was a matter of inches too small.
No problem. Two of them put their backs against the problem disk, and having been on the lifting course, pushed with their legs.
It worked!
The problem was that one of the guys got his belt caught in the Emergency Power Off button (which you have to pull to activate).
As they stood up there was a sudden silence around them.
It took a good few hours to recover.
They had been proud of their system availability, and had charts on the wall. Their carelessness was on display for a year, until it rolled off.
We had a nice office with a great view, but the only problem was this was over an outside corridor. When it was cold outside we got cold feet, so we brought in insulation, and put a carpet on top. ( We did wonder why lowly minions were given such a nice office - every one else knew about the office). We complained to site services who said - there was not a problem - and we were wusses.
A director took over our office (and the ones next to it). The first cold day she complained the floor was cold - and whoosh site services were round "oh yes there is a problem". They found that water from the rain pipe from the roof was leaking into the space under the floor. When it was cold - it froze - and the cold flowed up to the floor.
It took them a few days to fix the leak - and a few months to replace all the insulation along the corridor - having to work from the outside.
If you look at the fashion magazines showing how to furnish the room. The book shelves are always ordered, by height, or colour. (or an absence of books) These people clearly do not read their books.
Most people arrange them by author. I've got a row of books for an author by one publisher, plus another book in the series by a different publisher, which is a different size and colour scheme.
I remember "lessons learned" from several major projects which failed spectacularly. On of the common lessons was "understand and document the requirements before talking to vendors"
The requirements should be like "we want to ..." rather than "use this tool to..."
It sounds like the write back requirement was not documented.
My company found that the external site that sent most data to our site was the local football team which was sending MB/second!
When you are looking at a site, it can send you data, such as a 5 seconds clip showing a goal being scored, or a ticker tape.
Normally the remote site only sends data when the web page is active.
Their web page did not have this, so if the web page was hidden by other pages, it still kept sending the 5 second clip of the goal being stored.
Many (hundreds) of people would check the web page first thing, while drinking their first coffee of the day, and leave the web page there.
This was pointed out to the foot ball team, and they fixed it - and were very grateful , because their traffic dropped by a factor of more than 1000, and they saved lots of money
One of my bosses was in tech sales before he became a manager. I worked on the mainframe down in the bowels with the really technical stuff.
With my managers I used to say "there is a problem with customer ABC it is really technical... so I wont explain it" - and most managers were happy.
This manager said "I was in tech sales - so I'm not stupid, please explain the problem".
So I did - there is a timing problem between this widget, and that widget resulting in.... I knew within seconds that he was totally lost, but he kept nodding.
After I finished he said immediately "I'd also like to talk to you about..."
I heard he asked one of my junior colleagues about the problem, and got the answer "you don't want to know ... it is deep shit that no one round here understands".
He continued to think I was deliberately making it complex to make him look stupid.
It could be that they didn't actually know how the customers used the product.
With one product I was involved in, you could set a time out value tenths of a second. One customer used our product because the standard software only had a time out value in seconds!
They didn't use 80% of our product.
I remember going to customers and thinking "you are doing what! with it" - and going back to tell the test teams.
When I first worked nearly 50 years ago disk space was expensive, and I worked on DOS on the mainframe.
I was in the build group compiling a product. We had the build system and the live system.
I followed the instructions, deleted the old product, and started compiling the new code.
10 minutes later someone wandered round saying there was a problem.
I was working on the live system and not the build system.
My supervisor sorted out the problem and every one was happy.
They changed the instructions to say change this line.... to point to the build system.
Next week, I duly followed the new instructions and 10 minutes later someone came round...
I was building into the live system again.
My supervisor sorted it out - and changed the instructions to say "change >both< occurrences of..."
and that's one reason why I was moved to another platform with read only disks.
They also rewrote the instructions to include "what everyone knows".
I was involved (on the perifery) of a project to save money on a project. It was announced to everyone so we were all "part of it". Teams to produce plans.
The manager I was working with, called a meeting, and said "I don't want plans... I want actuals". We met up again a week later and had results like
1) because I always had to approve xyz - Ive set up a script to logon once a day and do it
2) we have all these old copies of software on disk - which we think no one wants. We've archived them to tape and freed up xGB of disk space
3) We don't think anyone uses these servers. We've powered them off. We can easily power them on again if anyone complains. We can virtualise them if needed.
All small stuff - but the first steps are the hardest.
A month after the big announcement the senior manager calling for the project to save money, asked for a review on progress so far.
Departments put up nicely written presentations about what they could do. etc
My manager put up one chart with the above points and said ( verbatim ish - it took less than 2 minutes)
- these are actuals. We are not doing any planning
- in any status meeting - people are allowed at most 1 chart
- the meetings now run to time
- we all go for coffee at 1000 - and raise any issues then - saving half an hour a day * 10 people
The senior manager took the hint and realised how much time (and cost) had been wasted doing blue sky thinking, and preparing charts etc.
He gave some ground rules
- No more overtime. Schedule your work better - or ask for help.
- I want to hear about meetings which are a waste of time
- Simple charts
- Get to know people in other areas who may be able to help you ( no more silos)
- I will be walking around the building - if you see me - tell me something (good or bad)
As a result meetings shrank (they had meetings but with 3 people not 13) and they found lots of overlap and so could save duplication.
Simple stuff but it worked.
"the definition of insanity is doing the same thing again and again and expecting a different outcome"
Sometimes I do something every day, and it works. Every now and again - it does something different.
And the opposite. Something is broken, so I start to debug it - and now it works.
Timing problems are a curse.
I did a day skipper course for how to manage a yacht. One exercise is "man overboard" which is a bucket, a float, and stick with a flag on the top. You learn how to turn the boat round, and get back to the MOB and pick it up. We all passed.
At lunch, we were anchored, having a nice lunch, I asked if we were to be taught how to get someone out of the water. No -- that's not in the syllabus.
So being in the med, in the middle of summer, I jumped over board saying "Man over board".
Every one burst into action
They threw me a rope. my hands were so cold I could not grip it.
Someone tied a few knots in a rope and threw it - again I could not grip it
Someone tied loop in the rope and threw it to me. Great - I put it under my shoulders and they tried heaving me out - they could not.
25 minutes later someone had the idea of using the mainsail winch - and I got out.
In real life I would have died.
If you have procedures you have to test them!
I was at a customer and some emergency stuff (key to the power room?) was locked in the key press. During practice it all worked fine. When they had a real emergency, the shift manager with the key was in a meeting with the senior managers. The senior managers were surprised when three guys burst in unannounced saying "we need the key - NOW" - and every one ( including the senior managers) rushed back to the ops room to help restart the systems.
Another customer had a wonderful document about what to do in an emergency - it was well tested, and online. After a power outage, they could not access this document because the server it was on was not active. Fortunately some one had a old paper copy which said ... start these server first... then you can access the document!
As a tester I got a reputation for breaking things. I was always impatient, and would multi task ( use two windows) when one of the tools was slow.
We had to use a new tool for entering what you had achieved over the year. Because there was no save button, I assumed, like Gmail, it had autosave. I submitted it to my boss who said it was empty.
I did it again...and again it was empty.
I contacted the help desk, who asked "did you press the save button?" me:"I don't have a save button"... "Did you press the help button ?" "I dont have a help button"
If you run the application full screen, all of the buttons appeared, if you run it in a narrow window, all of the buttons disappear.
With all of their testing they naturally used full screen, so naturally didn't find any problems.
I had a call with the development team, and mentioned the 50 "defects" I found. Such as
me:"there is an icon with 3 small circles - what does it mean? There is no hover text, and it is not mentioned in the help"
them: "It is meant to be obvious...",
me "well it has failed - so what does it mean".
them: "err we don't know"
me:"The fonts you use are not dyslexia friendly, and the fonts are too small and not changeable"
them "Err"
me: "Grey text on a grey background is not easy to read in a well lit area " ....
It is easy to start afresh. The hard part is migrating existing data to the new systems, bearing in mind it may be in different formats. Big endian -little endian, dates - images.
You might spend as much writing the conversion routines.
Some of the Nordic countries have very good systems. Can we buy/rent from these people.
Measuring may have no effective effect.
I was on a help desk. We had a slow-close policy. We would leave a problem open for a week, and close it if the customer did not call back.
We were told the number of open calls was too high - we needed to fix it.
So we closed the calls once the problem was resolved. The number of open calls dropped. If the solution didn't work, the customer now opened a new call. So now there were more calls.
We were told to reduce the total number of calls. So we reimplemented the slow-close policy, etc.
The solution was not the measurement of calls, but to fix the documentation/quality of the underlying code!
In the days when we had offices, there were some offices with wonderful views.... and our team was allocated them. We enjoyed the offices until winter, when we got cold feet because our offices were over an exterior corridor, and so the cold came up though our floor.
We reported it and were told cold feet is the price of a nice view. (We now understood why we were given the offices). Still we managed, we brought in carpets etc and were happy bunnies.
Until one Christmas when a senior manager wanted our offices merged into one big one.
Two weeks after our move out, the manager obviously reported the cold floor problem as there was suddenly a lot of investigation of the floor.
It turned out that there was a (drainage? ) pipe which was full of water which froze when it was cold... hence the cold floor!
We thought it was interesting that we reported it and were ignored. The seniors manager reports it - it is investigated next day.
I was at customer supporting a mid range system (it would support about 100 end users).
The boot sequence was usually
- press the IPL button on the CPU
- swing you chair round, and press enter on the operator console. The first display to hit enter was the operator console
- it would prompt "enter IPL parameters"
Except one day when I was there.
- press the IPL button on the CPU
- swing you chair round, and press enter on the operator console - nothing happened
we repeated it, and had the same result.
A few minutes a guy from the help desk wandered in saying "Ive got a guy on the phone who says his screen keeps saying "enter IPL parameter"
The guy was sitting at his desk, bored, so sat there with his finger on the enter key.
He was educated to only touch his keyboard when the logon screen came up!
My father was in the Royal Navy on ashore based ship. It was a ratings first day on guard duty.
One day this guy arrived at the gate, and the conversation went
Him: let me in, Ive left my badge in my office
Guard: No sir - no badge - no entry
Him: But I'm the captain of this ship
Guard: Sorry sir - I cant do it
Him: I'm the captain of this ship - dont you know me
Guard: Sorry - no sir -its my first day
Him: What's your name... .. now call the officer of the watch
OOW: Ah Hello Captain - what's the problem?
Next day the rating was summonsed to the captain's office. This usually meant punishment!
Captain: Well done for not letting me in. I left my badge behind, but if you had let me in, you would be here on a charge, rather than a commendation. Well done.
On my first trip to the US I had to present to a room full of developers. I had given it several times in the UK. After an hour I put up a chart saying "Fag break" - every one sat there bemused.
I said "Cigarette break?!" and every one got up.
I also learned on that trip that you park on the driveway and drive on the parkway.
At a conference where there was simultaneous translation from English into French, German and Italian, someone was going though a dump
"At offset Baker Dog Dog"... there was laughter as the French heard ".. Boulanger, Chien, Chien"
Having read all of the comments, it made me think about the software used to guestimate the spread of Covid. It had once worked... and then been added to multiple times, by many people, and after it was "improved for publication" it was still hard to follow.
They had problems like - run it twice with the same input, it gave different results. So there was clearly some problems ( because if you run it twice you should get identical results).
Manager's should be able to manage. They do not need to have been coders or tester - but they need to understand what their staff do.
We had one guy who was a crap coder, but a very good manager. His number priority was to protect and nurture his team.. Management wanted to promote him, but he said no... he didn't have the skills for management by spread sheets and upward facing. He liked managing teams who actually do the work.
30 year ago we bought our mother a laptop computer. She was living on her own and starting to get confused.
She loved the fish tank screensaver, of the fish swimming around.
I visited her once, to see her with a watering can, trying to find a hole to add more water, as it was a month since she had got the screen saver, and knew you had to top up the water in fish tanks.
We had a problem that when someone sat in front of a computer, the screen had problems. It was due to a floor tile squashing a cable. When you wheeled the chair over the floor tile - the screen went funny. When people came round to fix it - they didn't use the chair, but stood around the floor tile, and so it worked.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
