Posts by ColinPa 367 publicly visible posts • joined 23 Jul 2018
40 years ago, I heard about a senior operator training some junior staff, and he typed the command "PURGE SYSTEM ALL" on the product console, saying this is one command you must never , ever use. Then, like he does a thousand times a day, he pressed the enter/return key,
One outcome of the incident review was that they tightened up the authorisation checks for all commands - and many of us lost what little permissions we had.
A colleague did a customer audit/review, and recommended that the production console background was a different colour to others. A year or so later, someone issued issued a command on the wrong system. My colleague went along to do a root cause analysis, and found none of his recommendation had been implemented. When the feed back got to the board level, they found that there was an action "Get audit/review of system" Tick, this had been done. There was no action "Implement feedback",so nothing was done. Head's rolled, and several people were redeployed the next day.
We had an enterprise application based on a web browser which was designed to run full screen in landscape mode. Because it ran so slowly, I shrank it to about 3 inches wide, so I could do my day job while it was processing. Several times I entered lots of data, and assumed it was autosaved - because there was no save button. My boss came round and said there was no data.
it turns out that the save button was only displayed when window was 9 inches or wider!
I complained and got through to a developer who expressed surprise that I wasn't running landscape, and I had squashed the screen. They asked did I use the help button? This button had the same problem of not being visible. Also using the help button wiped out what you had entered.
I was asked to talk to a group of the developers ( and their manager) about the end user experience. I was allocated an hour, and we overran. I had 50+ comments, some were basic like "What does the icon of a dog mean? It has no hover text to explain". Why do you erase all of the data? Do not just truncate the screen - use scroll bars. Why is it so slow?
They suggested they could provide an education course on how to use it. I said providing this would be an admission of failure to design an intuitive system
Afterwards the managed thanked me and said it was very useful talking to a real user, rather than someone just testing the application.
I was involved in an international sporting competition. The computers were in a room under the main arena floor. With the additional visitors the toilets could not cope, and there was a leakage into the area under the area - about an inch or two a day. It was decided that by the time the competition was over, the "water" would still not be at table height. In normal operation, these computers needed no human intervention. The power was at waist height.
After the competition I think the computers were left there until the sewerage problem was fixed,then thrown in the bin.
I was in a hotel in Japan, and could not get the "local dial in" to work. I dialed to the UK ( expensive) and reported it. Next day I got an email saying the number does work, I tried it - it didnt. This pingponged for a week, when I finally asked the technician what number he was using. It was a different number to mine. He tried the documented number- it didnt work because it had been misprinted.
Someone told me that after their request for upgrading the infrastructure got turned down because of cost. They fixed the bean counters, by turning off one of the old server - which supported the bean counters and their systems ran very slowly. Cue lots of cursing from the bean counters. After 2 days they turned the server on again. The bean counters said "that cost is xxxx thousands points in lost time" The sysprogs replied - it only cost y thousand to replace it... so if we could have upgraded the server it would have saved you all this money.
I have a friend who worked in a secret area where they had complex data processing. The found the best people for this work was "weird" mathematicians, usually on the autistic spectrum.
HR got involved in the hiring, and only "well rounded" individuals who fitted into the "corporate ethos" passed HR. My friend said these people were very nice, but they did not know the difference between a theta function and a zeta function.
My mate tried to give HR some guidance. If they wear odd socks - put their application form on the top of the pile.
We had someone who did a year with us as a sandwich student. He was great. Next year he came as part of graduate recruitment. Technically he got all the approvals. But HR turned him down - "he was very quiet". They thought he would not fit in!
My manager "spoke" to HR and they changed their minds.
He turned into an exceptional technical leader.
A friend worked for a shipyard in the computing department(may years ago). They advertised for a sysadmin/sysprog, and of course it had to be advertised company wide. It was surprising how many welders etc turned up for the computer aptitude test, looking for a nice easy job in the warm.
My sister was a nurse. She said the "new IT team" came round to see how people used the system. Because the wards were very busy, the new IT teams had more time with the back office team.
The result was the system was great for the back office team - but useless for the ward staff. As my sister said having the patients address on the front screen was fine for the back office - but useless on the wards. She had to page down 3 (slow) pages to get to the "on the ward" page. 10 seconds for every access to a patient's details soon adds up.
NEW EMAIL SYSTEM
She said they were also going to change the email system to a different system, with different format email addresses. When they asked how they converted their address books, they were told you contact all the people and ask for their details... then you update your address book with the new info "overnight". My sister said I have nearly 1000 contacts in my address book. I use 200 regularly. The management decided to put the project on hold - then cancelled it.
When I first started work, we had a 360 DOS/VS system. Our development team had 2 disks. One live, and one build. After the build was successful - they swapped disks.
My job was to compile the product onto the build system. My second day in the job, had people coming round saying "its stopped working". Whoops I had built into the live system. My team leader came and discussed it with me, and agreed I had followed the written instructions - but there was one line missing. "Mount the build disk". He edited the instructions.
Next day the same problem happened - I overwrite the live system. My team leader came round... and agreed that yesterday's build disk was today's live disk, so the documentation should be clearer. He moved me to a different area where I could do less damage.
For most of the projects I worked on the requirements was one of the first deliverables, before you staffed up, because you may not know what skills you need! They are 6+months in and do not understand the requirements.
A friend led a government contract providing systems across multiple departments. He called a meeting to present back to the stakeholders what he thought the requirements were. All the requirements had been "cut and pasted" from the departments into one document. Many requirements were inconsistent.
His first question "I assume you all know each other?" quickly showed that the various departments had never met.
He got every one to say who they were, and what they wanted out of the project. He then sat back refereeing the discussion. People had no idea of scale... "we want a list of people who...." "Really - that's about 2 million people" "what will you do with the list - ring them up?".
My friend said that the final requirements were much simpler, as most of what they wanted was already provided else were - it just needed some glue to join the bits together.
I think half way though the project was cancelled because they realised they didn't need it.
I was in Japan in the 1990's when hotel connectivity was to plug the phone line into the back of your computer. This didnt work because of a dialler problem ( which allowed me to connect to the corporate network) but there was a fix for it. My problem was how do I download the fix to be able to download the fix. This was late Friday night and I was working a a customer over the weekend.
I took the subway (1 hour) to our corporate building, but I could not get into the floor because my badge was not on their system, and there was no one in! I found there was a corporate WiFii signal if I held my laptop up. I downloaded the fix, went down the ground floor, and borrowed a telephone line from a security guard (who spoke no English) - and it worked!
These days with hotel Wifi - it is so easy.!
One of my colleagues had a habit of if anyone gave him some confidential paper work, after he had locked his desk, he would put it in the bin. The security guards who checked the clean desk policy never looked in the bin. He was always first in the office, before the cleaners, so just got the documents out of the bin.
Someone told me that they worked on a project where the development was done in one country and the testing was done in a different country. Due to the executive announcing the customer date before development was quarter done, it was pretty chaotic and corners were cut.
The installation team followed the process of getting a clean machine and trying to install the product. It always failed. They executive blew his top and sent his top expert to "SHOW THE INSTALLATION TEAM HOW TO DO IT". The guy got off the plane, went directly to the lab and tried to install it. It didn't install. He spoke to the development team - they hadn't installed it either - they just replaced the binaries every day and were careful not to turn the machine off. I think no one had actually been given the job of writing the installation process, people just used some scrappy notes.
In the development team, because the project was running late, they were asked to resize the work. Now they knew what they were working on, instead of some chart-ware design, they resized it properly - and said it had gone up from 3 months work to 9 months work. The executive blew his top as this was not the answer he wanted to hear. He wanted them to resize it again! but someone told him - every time they resize it - it costs 2 weeks work. Do you really want to ask them?
We tend to focus within a cone. So a 2ft square display 2 feet away works - so to see a 10 ft high "window".. I would need to be 10 ft away and would need a bigger font size, and thus get the same quantity of information (if the VR technology was up to it)
I was on a customer site when the fire alarm went off and they evacuated our building. There was a builder's shed in the car park which was in flames so it was a real fire. I believe there were gas cylinders in the shed - so potentially very dangerous. There were several buildings linked by corridors - and in order to get from building 1 to building 3 you had to go through our building. Despite fire wardens trying to stop people going into building 2 - people pushed their way past.
The fire brigade head officer happened to be in building 2 and asked why people were transiting through building 2.
Apparently the fire officers report was interesting reading. I heard there was an action plan along the lines of education ... "If the fire alarm is ringing. Leave the building. Do not go into the affected building to collect your keys, or to transit the building, or to go an collect your coat, or a cup of coffee... If someone is injured or killed as a result of your stupidity, you will be liable."
Ive worked with many customers who ran two systems in parallel. You syphon off the data from the old system to the new system as it comes in... Everyday you compare the reports, and check they match. You do not go live until the data matches. It is slow - but better than debugging the live system.
I remember being asked on Thursday, that someone needed this product installed as it was needed for a demo on Monday afternoon. This was a mainframe system, and the products came on tape.
We paid for a motor cycle courier to deliver the tape - and it arrived Friday 4pm. I spent Saturday installing it, and getting it to work.
I put an overtime claim in to the requester's manager.
I emailed the requester saying it was installed. I didn't hear from him, or how the demo went, so I wandered over to his desk on Tuesday morning, and he was very very grateful for the work I had done - the customer's were very happy etc.
I then asked him which userid and password did he use? "um er, er um Admin!" and what password did you use?
He then came clean that they had not actually used the product. They found this out on Friday but had forgotten to tell me.
His boss queried the overtime, I explained what happened, and he rolled his eyes.
I heard about the guy whose code had the highest number of abends/sigv. He was leaned on to "fix it", even though he said his code was innocent.
To please the bean counters, he added code to value check every parameter passed to his routine (which doubled the path length). Anything that failed got passed back a "error in parameters" return code.
Suddenly his code was getting no problems, but every one else's code was reporting "error in parameters".
He compromised by providing two versions of the module. One for "friends" who passed in good data, and "general" for every one else.
Someone said that he job was "The devil's advocate", and it was his job to explain/demonstrate why things will not work. He explained one project which relied on having a mobile signal to work. He took a holiday in the wilds of Scotland, and came back reporting that he only got a signal 50% of the time, and nothing up in the mountains. They changed the project as a result of this and some some parts of the project they used satellite phones. He would set up his test environments in a tough area (eg weak or no signal), rather than the middle of a city which had good coverage. His philosophy was "find the bad news as early as possible - it gives you more time to fix it (and factor into the price)".
Having your data encrypted means someone who does not have the keys cannot read it.
If you do have the keys then you can usually copy the data (or photograph it).
I have data on an encrypted disk. The data is visible to me because I had the key. I can easily copy the data to a USB. It is going to be a challenge to make this secure.
Our printer just printed what was sent to it.
There was a sign saying "Please file any output in the relevant slot. Eg Output for Jones goes in 'J' ".
What happened was all output was filed under "A".
Some wag put up another sign "For those of you that do not know your alphabet there will be a class next week - but I doubt if you are able to read this"
Another way is to expire all the password from the group of unhelpful people - and see which userids get their password reset.
Someone else looked at the cost of some servers, and decided to charge back to the managers. The manager getting a bill for "£50,000 for support of unused server" quickly took action. For those that didn't take action - a month later the manager's manager got a copy of the bill ( £500,000) for support of unused servers.
Someone told me of when they worked in a small software company doing pretty advanced stuff. You could get fixes onto the system and incorporate them without having to restart the system. For 6 months the development team fixed all their problems this way. It was fast, it was slick - and looked great.
At a moment's notice they were asked to do a demo for an important customer. The unlucky person created a memory stick image, and rushed to the conference room with his laptop. The system almost started. When the boot got to their code, it fell over. For the previous 6 months their server had not been rebooted. He sat there trying to fix it, when the manager came out and said "We are over running, do you mind just giving a 2 minute description rather than the demo". Suddenly all was calm, and he said "that was fine by him".
We had one guy in the test test team was great at finding bugs - from the simple to the most complex. We were asked to write up our test strategy for a prestigious magazine. . Based on his test philosophy the article was like ...
"You have been asked to test this new car. Go and visit a local farmer. Go at speed down the dirt track, aiming to hit all the potholes. Whoops you've got a puncture. Raise defects for the following. 1) no instruction book on how to change the tyre. 2) No spanner for loosening the wheel nuts. 3) cannot see where to put the jack. Once those are all fixed - do the same except drive it in reverse - at fast as is safe. Cross a muddy field - raise a defect because you cannot attach a towing hitch to pull you out. Run the car till it runs out of fuel... then add some (from a can) and check it continues to work ok. Give the car keys to the kids and say they can do anything but not drive it. By now the car is a bit scratched, take it in to the workshop and make sure they can do a paint job - what no paint of the right colour ? Another defect. Park it on a steep hill - not in gear - so it is just held on the handbrake.
Once you have done these basic tests - get your granny to drive it.... she is not very tall - and you find she cannot reach the pedals without the steering wheel pressed against her chest.
We submitted it - but it was rejected as being to "simple" and understandable. One reason was there were no graphs or complex equations.
One guy I know almost got an award for negative defects per lines of code.
He put in a thousand statements like "x= 1;" knowing the compiler would optimise these to one statement.
As a result his defects per lines of code was really small. The bean counters algorithms could not handle this (due to rounding errors in their code) , and and some times produced negative defects per lines of code.
He was nominated for an award till someone queried the results, and looked the code.
His argument was he did this to give him time to write the really difficult code without being chased by the "project managers" (and wasting his time).
I remember being in Sydney for the 2000 Olympics. Some of the Americans said that that thought the local drivers were very polite and old fashioned. When the American's made a mistake on the roads, the locals would touch their forelock as they used to when the local squire came past.
I had to explain that touching the forelock could be taken a different way ... known as Richard Head!
After a customer moved from main frame to "commodity hardware" (lots of x86 boxes) I got dragged into discussions about a performance problem.
I could see from an I/O trace that some I/Os were taking 100 ms... the rest of the time it was below 1ms. I reported this to the virtualisation team.
The virtualisation team said "on average the response time is below 1 ms across all the machines. So not a problem. It must be a storage problem."
The storage team said "Across all our storage the average response time is below a millisecond! - not us".
This had been going on for about 3 months. Eventually the customer took drastic action and turned off auditing to disk, and other things that did I/O.
After a lot of analysis it looked like there were some hot files on one disk. If many virtual machines all did something at the same time - you got I/O delay.
All the vendors were to keen on not looking bad - rather than resolving the problem.
A year or so ago (during the tail end of of covid), our local theatre was putting on one of those plays where every thing goes wrong. A theatrical flash went off on one side of the stage, and the genie appeared on the other side of the stage half dressed ... ha ha ha ... until the fire alarm went off.
We all dutifully trooped out, asking was this part of the play? We were half out - when we got told we could go back in ... so in we trooped - and we were told we >had< to evacuate.
If the fire alarm goes off and the fire brigade is summoned, you have to evacuate. etc.
Fortunately the lighting guy was a fireman and had been paged and could cancel the fire engine. We could then go back in... etc
The problem was caused by having additional doors to the theatre open for covid. Normally the over stage detector is disabled for an hour which is fine... but the wind through the open doors wafted the smoke into the corridor and so set of the alarm!
We had a tool which "helped" us decide which visa we needed for different countries. It wasn't very good. It said a UK citizen needed a visa to go from the UK to France, and could get it from the French embassy in Washington!
It also said I needed a work permit in China if I was doing any "education". This work permit takes 3 weeks + to get, and you need an address in China. As I was flying in Saturday morning - doing a "on support" Saturday night and flying home Sunday - I didnt have time to get an apartment or apply to the local police station which is only open Monday to Friday!
After weeks of complaining about the tool - they changed the tool to say "UK citizens do not need a visa to go to France"!
Every one else ignored the tool!
I remember organisations (40 years ago) getting a small mainframe, then using Bureau services, then bringing it in house, for control, availability, security and cost. Then "cloudifying it, then bringing it back in house.
Another example was "the main frame is expensive - we can do it cheaper with departmental machines". Then realising this simplistic view does not work - you need multiple machines for availability, you need to put fixes on it!, you need extra staff to support it - you need to do security audits, and backups, you need to manage (and pay lots of money for) software licenses. You need... a whole departmental IT sub-department. You then find your box sits there most of the time doing very little.
You find this is very expensive, so this then swings to having a central site manage all of these aspects and making much better use of people, and resources. As a result to use the centralised services you have to put requests in which get prioritised and there is a few weeks delay, so you get your own machine... or put it in the cloud - till you find the cloud is too expensive,
I was involved in supporting a large sporting event, where email addresses were provided for the participants.
Initially they put mail filters on to stop "bad" emails getting through.
There was a Dutch speed skater with name like xxxx le bombe ... who got no mail!
They they found the simple statement "We have your daughter" was both a very harmless phrase - and very sinister thread. Eventually turned they email filters off as they could not adequately filter the emails.
We were a development lab. We had some building work next to our machine room, and a digger dug through the main pipe from the air conditioning units to our building. It took a week to fix (getting the replacement pipe was the problem)
So no air con - and all but critical systems were shutdown. Also our building had no windows (why do you need them when you have aircon) so the offices soon got very warm with all the warm bodies.
This was before the days of being able to work from home. Some managers were enterprising. Some held meetings in the nice outdoors, under the trees. Other managers said "please work from home - and review the documentation". So people took the paper manuals and reviewed the stuff relevant to their area. Lots of stuff got pruned, and there was a lot of new content.
So overall it was not too bad an impact ( besides the sweaty bodies)
re manglement scoring 5/5.
We had a guy like that, so we hatched a plan. We had a deep problem which we could not resolve, so we assigned it to one of the "expert managers" with a 5/5 in the area, and made sure his name was on the charts for action and resolution. Suddenly he was not keen to point the finger at us. We eventually got him trained
Our management wanted to know the skill level of people in the department, so a questionnaire was created, which we all had to fill in. The questions were along the lines of What is your skill level in Windows/Linux/Mainframe
0 - I know nothing
1- There is a lot I do not understand
2 - I know a lot about it
We were told to answer the questions - and not to be too clever.
I had worked on the mainframe for over 30 years - and knew a lot about it. (People came to me for help) - but I knew there was so, so much I didnt know, so I put my self down as "1 There is a lot I do not understand"
The results were interesting.
The "experts" all came out as "need to develop their knowledge". The people who knew how to logon (and not much else) came out as "experts".
At first management were very happy, as they had some data. Some of us techies were asked to review the report and made comments like "give them a kernel dump, and they would not know where to start".
Us techies then rated the other people in the department, and the results were completely the opposite.
Management then realised they had asked the wrong question. They should have asked "do you have the knowledge to do your job" ( ie do you know which buttons to push) - but they gave up and this survey was never seen again.
We learned that you get what you ask for - so make sure you ask the right questions.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2023
