Posts by ColinPa 406 publicly visible posts • joined 23 Jul 2018
A friend's mother had a laptop. When she changed her password - she wrote it down in a file on her computer. My friend had given himself a userid on the machine, and managed to find the file, and retrieve the password. His mother loved crossword puzzles. The file was called something like Kyber.txt Because everyone has heard of the Kyber-Pass Word.
I did a review of a system, and one of the charts had the line
"You are a very lucky company".
Initially the management very chuffed till I explained what it meant.
You have been skating on thin ice, and not fallen through so far. You had made changes without understanding the impact. If there had been a problem, you could not have backed out the change. As I said - you are very lucky you have had no problems.
In the days of 3340's which you could physically pickup and mount/unmount (and looked a bit like the starship enterprise), spinning disks etc.
One of our testers who was an operator in a previous job, had had problems with the disk containing the master database for the banks customers. He called over the senior operator who said.... we had better try it on a different drive in case the drive is suspect.
It didn't work there either - so it must be the disk. The got out the mother disk. Yesterday's database is copied to a different disk and the batch update run to make today's database (so Mother database begats today's database).
That didnt work either, so the senior op got out the Grandmother disk from the manager's cupboard. Mounted it - and it didnt work either.
So they phoned the manager who said "that's ok - just do not touch the grandmother disk".... "Ahhh too late - came the response".
There had been a head crash on the original disk.
Mounting it on a different disk drive damaged the heads of the second disk drive.
The mother disk was corrupted by the damaged heads.
The grandmother disk was then damaged by the damaged heads.
Fortunately they had a copy of the database which was only a month old, and could reapply the overnight changes which took about a week to do.
And that's when the tested decided to join our company where he could do less damage.
My father was in the Royal Navy, and they had some kit which didn't work. He suspected the suppliers were going to pass it on to some unsuspecting person. He had a junior rating scratch "condemed" inside the machine, and left a phone number.
A month later he got a phone call saying "we've just bought a new .... and when we opened it up, it had "condemed" - and your phone number.
I spent two weeks out in Asia as part of a team who were fixing one or two major problems at a bank. We solved lots of problems. At the end of the last Friday afternoon we presented to management about the problems we had found, and how we fixed it.
We finished the presentation and were heading towards the bar for a few beers, when someone timidly put their hand up, and said "but you haven't actually fixed the problem we asked you out here to solve". Coats off... laptops out.
It was a "simple" configuration problem. 10 PM that night they put the fix into production. It worked. Back to hotel at midnight, bed and depart 0600 next morning... so no beers.
From this I learned that it is worth walking round the techies at the beginning and getting their view of the problems. A common comment was "We've told management what the problem is, but they don't believe us"
We were working in an old building, and someone wanted to hang a few pictures on the wall,so had a hammer and some 4 inch nails.
One of the old hands told us to be careful. He said that the high voltage cable wandered around. It should go up and down, and be in a protective tube - but not in this building. Sure enough, we pulled the panel back from the wall to have a peek, and there was a cable just wandering around (in coils), not secured etc. It was not done by electricians, but by some people who wanted a "quick fix" until it was done properly.
The old hand said don't look under the raised floor - so we did. If people wanted to add new connections - they just added new ones - and left the old ones behind. It was 9 inches deep in cables.
I remember one guy who thought he was top notch, but was all talk. He would "work" long hours to get the job done (and claim the overtime) and told every one how hard he worked (He spent more time talking than working).
At one status meeting he was said he was half way though a piece of work, and it would take him only another 6 weeks to finish it. I spoke to his manager saying "You can either do this all by hand, or a write script to do the heavy lifting" I wrote a script in about a day.
The manager had a conversation with the employee. The employee decided that retirement might be a good option. It was that or be exposed at a status meeting.
After he left every one became more productive, because they didnt have to listen to him!
I was on a week's team building exercise. One morning we were split into teams and asked to discuss a topic and present on it.
Four people including a guy called John were given the title "working as teams".
When this lot came to present, there were two presentations. One from 3 people, and one from John both called "Working as teams". John could not see the irony of the situation.
Two similar incidents
A Canadian bank had two data centers - one on the east coast and one on the west coast. They wanted to use mirrored disks! The only problem was the network latency, so they moved the data centers closer to the middle of Canada, far enough apart to provide isolation.
(I heard of a bank in the US which had two data centers, one in North California, one in South California. Which was fine till someone pointed out the San Andreas fault line went through both data centers. They moved one east)
The other incident was someone complaining about performance of stuff across Asia. They said when they tested it - it easily out performed the requirements. When they rolled it out - the performance was terrible. I was one of a team who was sent out to help. In their testing they had two sites, but "one site" was on the second floor, and "the other site" was on the floor above. The network distance was about 30 ft! (Not the 4000 miles true distance between sites).
When we probed the requirements, they did not actually require each transaction to go to the remote site and back in under 10 ms. They could "batch up" the work. Do 1000 transactions, then check the status of the first transaction etc. Problem solved... every one very happy - especially as the CPU cost was reduced significantly.
I was on site where other people were trying to get a connection going. The remote end had configured the long random string password, and raised a ticket, so our end could set the password on our end.
They guys cut and pasted it, and it didn't work. Eventually they said, let's type it in. So one guy read it out, the other guy typed it. When they got to "O" the typist said is that an Oh or a zero? They picked Oh and it didn't work - they repeated it with zero an it worked.
The ticket raiser has typed the wrong password in.
From this I learned that every change should be cut and paste, and not typed.
a) It saves time (you do not have to think)
b) You can test it before doing it in production and be sure that what you are doing has been tested.
c) You have an audit trail.
I liked the version which mis quoted this.
In the grammar test James, while John had X had had Y, Y had a better result on the teacher.
Now replace X with <had had "had", had had "had had", "had had" had had >
and Y with <had had "had had", had had "had", "had had" had had>
You now get about 30 had's in a row.
It's turtles all the way down.
In the 1980's I heard the story about someone who wanted to get some more big (6ft high 4 ft square) physical mainframe disks into the machine room, but there was not quite enough space on the floor.
If they moved those disks 6 inches they could get the new ones in. So while the disks were in use (!) they leant back against the disk, and pushed ( with their legs as they had been taught) but could not budge it.
Unfortunately, one of the guy's trouser belt caught in the Emergency Power off. So when he stood up, the EPO was pullled and the whole bank of DASD lost power.
This took down the whole banking system - whoops.
Someone said that the best way to get the right things was to get the bosses to have to use it.
So I suggest the top brass responsible for this stuff get dropped into the middle of Australia or Africa, or Siberia and have to use it for a week in full battle kit.
How heavy are the batteries? Is there any WIFI? - No WIFI because of the terrain is hilly and there are dead spots.
I remember having to use a face mask in a hot dusty environment - one problem was the sweat would collect and slosh around inside it. Periodically you had to take it off, and pour out the sweat.
How is this different from all of the other frameworks the government has issued?
Have they looked at how other countries have done it, so they start from a working system?
Have they factored in all of the "lessons learned" from previous failures?
Have the pigs been fed and are they ready to fly?
Ive just experienced a similar but different problem. We were in a drama competition, where groups put on a play, and an adjudicator gives marks for presentation, acting, technical etc.
We had a play and a critical part was projection on to the back of the stage. The projection worked the day before.
When we tried it with out play, we found someone had hung a black cloth in the auditorium - blocking the projection.
Our lighting man got his own projector, plugged it in to the mains, and computer and it would display the set up screen, but not talk to the computer. He tried everything. He finally used a different extension lead thinking it will make no difference, and it suddenly worked.
The problem was that the computer and projector needed about 5 second to sync up. The extension lead had a dicky plug and would drop power for a millisecond, every second or so. As a result, it never synced up
This took him 60 minutes to find - all the time we were allocated to set up the technical stuff.
I had to visit a customer. The technical team were great. They had an idiot for a manager. He had been an excellent technician responsible for cabling.
The offices had glass walls, (some were opaque). The boss did not realise that the wall behind him acted as a mirror, and we could see that he spent a lot of time playing Ma-jong on his computer.
We presented to him, (and his peers), and his only comment was to ask us to change the header page, as the colours were wrong, and there was a typo.
This was a classic example of The Peter principle (https://en.wikipedia.org/wiki/Peter_principle)
"which observes that people in a hierarchy tend to rise to "a level of respective incompetence": employees are promoted based on their success in previous jobs until they reach a level at which they are no longer competent, as skills in one job do not necessarily translate to another.["
IBM's mainframes (VS1, MVS, VSE, the original DOS, zVM) running on 360/370/390 hardware were not written in C. 40 years ago they were written in Assembler, and PL/S (A pl/1 syntax like language). This evolved into PLX which is used these days.
These days application type stuff like web servers (and TCP/IP) may be written in C and Java. Even if you wrote in "Metal C" which was closer to the hardware, you still have to drop into assembler to do the hard stuff.
C has a rich library of functions like printf(), but the assembler and PLX had equivalents but they were not so easy to use.
I recognise these words. They are usually used during basic functionality, running a single task.
It they had said, "When running this at expected peak workload + 50%, with x thousand concurrent users" it performed well. I would be more impressed.
The problem with large systems come down to concurrency and locking/latching
My sister was a nurse in the community. Before she started work each day she had to fill out where she was going, what she was expecting to do, and how long it was expected to take.
When she got back she had to enter on a DIFFERENT system where she had been, what she has actually done, and how long it took.
The first month she carefully filled in the data.
Then she started adding comments like "Does anyone actually read this stuff".
She tried putting garbage in but the system checked for valid numbers, and fields had more than n characters. So that was easy.
When she retired, she said that no one had every spoken to her about the data, despite being told it was needed.
Go back 40 years .... I was told nth hand of a product running on the mainframe, which had a problem which was very difficult to solve. He put some "temporary" debug code in along the lines of
* We can never get here. But if we do we are in deep trouble. Phone John on 01962......
WriteOperator "We can never get here"
Normally all the code was inspected before it went into test; to stop bad code and comments from escaping. Temporary code was not reviewed as it was not shipped.
6 months later after the product had shipped the phone rang - it was a customer "is this John ? We've had a problem, and the source says to phone you..."
I remember going to a conference in the US. One of my colleagues, his first trip abroad, did a couple of back to back sessions logging on to the systems back in the UK.
These were the days before wireless where you unplugged the cable from the telephone, and plugged it into your laptop.
All went well - till he came to check out of the hotel and had a phone bill of thousands of pounds (more than the cost of his whole trip).
When he got home and tried to claim his expense, his manager paid (after lots of grumbling) and asked "Why didn't you use the AT&T freephone local number". No one had explained to the poor guy how you work internationally.
On Friday the "tools" team said it would take about 2 months to date some data and produce reports using their tool.
I worked Saturday, loaded the data into DB2.
On Monday I said "this is what I expect... " and showed them reports and graphs.
They said "What's DB2?" They had spent 6 months + working on their tool
Having 100% of the automated tests working, just means that known problems have been fixed.
We upgraded our machines, to have more and faster engines, and the tests found more problems, such as timing windows.
If you do not build in randomness the outcome is predictable. If you introduce randonmess, such as time between events, number of tasks running, size of the peaks and troughs etc, each run becomes unique. This makes it a devil to rerun and check any fixes.
Running the "golden path" (where all test work) doesn't prove much. You need to go "off piste". You do not test drive a car just by driving it up an empty motorway. You go down side roads, up farm tracks(at speed) etc.
As a product manager said to me "We are not interested in hearing that all tests have run successfully. We want to hear where it broke, why it broke, and how to fix it. If you haven't broken it - you are not pushing it hard enough"
I remember a site where a digger managed to fracture the aircon main pipe between DP room and the aircon plant. Having dual power supplies etc didn't matter if the kit gets to hot to use.
I also heard in France where a company had two external network. One went via the north of France, and one went via the south of France. Which worked fine, till the telecoms company put a major backbone across the middle. So the northern network went north, then south to the backbone, across France, then north, then south to their office (and the mirror image for the southern network). This all worked well until there was an outage on the backbone, and both networks were impacted.
In the days when Java was still new with all these bright keen youngsters, I remember going to a presentation on it, and someone saying that it supports transactional recovery (2 phase commit)! (whoo hooo!).
Some old geezer in the back row, said "I agree it is a hard problem to solve. II took us some time to solve it 20 years ago on the main frame, why didn't you come and ask how we did it, and save yourselves the pain of rediscovery"
A friend of mine worked on a major UK Government project and had to manage all of the changes that came in "after the spec was agreed". He said they charged money even to consider a change, and there were many changes. There were often had contradictory requests from left hand and right hand of government.
He partially solved this by organising a day with all of the stake holders in one room to discuss the high level implementation. He said he was amazed that his first task was to get people to introduce themselves to the other attendees. Many of the people had not met the rest of the team, and had no idea they existed, even thought they had, in theory, collaborated on the original requirements and come up with the common list of requirements.
My friend had a list of questions like "what happens if someone changes their name, or changes sex",but they did not get to these questions, they got stuck on the first chart "stake holders and their interests".
He said that they had several iterations of these get-together meetings, and gradually converged on the requirements and what was actually needed.
I had to visit a customer about one project, and was asked if I could come to a meeting on a different project to listen in and offer any suggestions.
One high level problem was the mainframe could process about only 20 requests a second - and they needed over 1000 a second. My product produced statistics and could show the rate was very low (and therefore was the culprit).
The manager of the project was shouting that it was not good enough and he was going to recommend that this pile of manure should be taken out and the mainframe be replaced by a Windows server. "You have 24 hours"
I asked questions like "do you know where the bottleneck is?" "what evidence do you have", and the answer was "20 years experience".
I worked with the mainframe guys and looked at a trace. The mainframe was almost idle. We saw that data flowed into the mainframe every 50 milliseconds - which matched the 20 requests a second. The problem was clearly upstream of us.
I left the guys to it. Next morning there was a very tired. but happy team of mainframe guys.
They found the upstream server was running on a 10 year old server running Windows. It had not been patched for a couple of years, and no one officially knew about it!
I do not know the gory details of what happened, but next time I visited the shouty guy was gone, and more work was moved to the mainframe. The team were happy because lots of problems disappeared, and they met their service level agreements.
I was in Australia for a major project in several stages. Rather than come home, between trips, I got my employer to pay for me to pay for me to do some "good will visits" so I had a 3 week "vacation" with 6 customer visits paid for by my company!
I went to one customer and spent the day with them. I think it went well, in fact I got taken out to the local bar by the local systems programmer who asked me details how it worked internally. I explained in general terms (missing out all of the confidential information) and we covered many napkins.
Next day I got an urgent email from the head of support asking what the hell was going on.
The guy had gone back to the office raised a sev 1 problem and slagged of our product saying what a crappy way to design it, and had we thought about...
I had to write back and say "I explained in general terms. When I said the product searches for the item of interest, you assumed it was a linear search. In fact we have a hash table. Where you assumed we used the expensive xxx function to lock, we have a 8 instruction "check and lock"...
I got to meet up with the local team from my company who said "ahh - we should have warned you about the guy. He is very bright, and assume he is always right".
Someone told me that if his software detected a problem, it would package up the relevant files (FFDCs logs etc,configuration files) so they could be sent to support, and so be sure of getting all the files.
This worked great. One day they had a very large file containing lots of hidden files. When they saw the titles of these hidden files, and looked at the content of one of the files, they could understand why the files were hidden! The originator was smart enough to hide files, and thought that burying them deep in a "operating system" directory tree would mean they were invisible to every one else.
The support team hatched a plot.
They sent an email to all the users saying a bug had been detected, and affected users would be sent a fix. They sent the "fix" to the user, and someone went round to help him run it. The fix just deleted all of the naughty files. The "fix description" was along the lines of "remove files which should not be on the computer"
They also put in a sneaky fix which checked the number of files in the directory and reported "illegal files found" to the end user and support if there were unexpected files found. They were surprised how many alerts they had.
I was periphery involved with a new product. The first beta code drop produced messages like "EARLY CODE: NOT FOR PRODUCTION USE. VALID UNTIL xx/yy/zz".
The customers signed the terms and conditions saying this. 6 months later we shipped the proper product. Customers had to order the product, pay for it and install it.
Two months later on xx/yyy/zz we had a panic call saying "production is down. It will not start. It is producing a message 'EARLY CODE: NOT FOR PRODUCTION USE. THIS CODE HAS EXPIRED'. What can we do?"
The answer was "you need to order, and pay for the GA product. These checks are only done at startup. If you have another instance running. Do not shut it down."
The customer paid his money, and the tape (this was pre internet days) was sent by motor bike courier. Normally tapes were meant to acclimatise in the tape library for 24 hours. This was skipped, and the tape rushed to the machine room. By mid afternoon they had it up and working.
On the mainframe we had to worry about sources files in ascii and EBCDIC, and there was a setting which set the code page on the file. The various zip/pax commands did not always work.
Our simple instructions gave a nice little process for doing the work.
Some "clever" Windows expert thought 'I can do this with less typing and make it "quicker".' as a result his data was always in the wrong code page.
My colleague was on the phone for an hour trying to work out what the problem was.
The end of the conversation went
"Do step 1"
"Ok I've done step 1"
"Are you sure?"
"Yes"
"hmmm - what was the response from the command"
"xyz"
"But that's not the response from the command. It looks like you used the abc command"
"Well it works just well"
"No it doesn't it. Follow the instructions"
"OK Ive done step 1"
"Now do step 2"
"what response did you get?"
"That's not the response from the zyx command"
Loop:
Off topic...ish
A shipyard in Scotland is trying to build two ferries. They are over budget and late. The latest problem is the data cables are not long enough.
The cables were long enough when they were ordered a couple of years ago, but someone decided to move the displays and instruments, so the cables are no longer long enough. Swapping the cables around is not an option, so even more will have to be ordered.
The obvious answer would be to move the displays back again - but that's another can of worms.
Talking about trips. I had a colleague in sales and marketing in the US.
He was just told "take his laptop and be on the corner of Washington and Third" at 0900. He wasn't even told the customer name. A black limo picked him up - he could not see out. They drove around with many turns, and eventually went deep underground.
He got out and was met by a big marine with a gun at the ready. "Where am I ?" he asked "Don't ask" came the reply.
He was taken to a room - it was empty; and a voice said "please do your presentation".
He did his presentation to an empty room and a voice said " thank you - you can go next door for coffee, we'll recall you if we have any questions"
He went for a coffee - but needed the toilet, so he asked the guard who escorted him to the toilets. The cubicals had no doors and the armed guard watched him do his business.
He was called back to answer a few questions and was allowed to leave; the same limo; it drove around for half an hour and dropped him at his hotel.
He said it was the scariest presentation he had ever done.
"In April and June last year, new requirements from the HR department continued to arrive after the main software build was complete."
Sounds like they didnt know what they wanted till they saw what they were getting. It sounds like "I I know it is what I asked for, but that isn't what I meant"
I sat in the tail end of a post mortem in India which was trying to find out why a production system had been shut down.
The site "expert" (arrogant, would not be told) was defending his corner. An external expert said "6 months ago we did a review of your systems and one of the top 3 recommendations was to enable the screen colouring; red text for production; Yellow for test. Why wasn't this done? It is a one line change"
The site expert tried to bluster his way out blaming someone else.
A humble (but skilled person) said "excuse me, we did create a change record, but it was never implemented....".
We had a comfort break, while management found out the status of the change record...... all the recommended improvements were waiting for the site expert to approve!.
We didn't see the expert again... but someone else, a team leader (rather than an expert) took over and within a week all of the recommendations were in progress, and calm and confidence spread throughout the team.
I remember the new manager of the world wide tech support team, offered to spend a week on the phone so he could understand the job, and what he had to manage.
At the end of the week, one of his comments was "you are not being paid enough to take the crap you get". He managed to get the "bonus" pool to be spread over the support team, rather than the guys who wrote the code in the first place!
We found that people who spent a month of the tech support, ended up as better developers. For example they put trace points into the code at critical places, messages were very helpful, rather than "consult your systems programmer"
Our software would return a code equivalent to "disk full".
Programmers would check for errors they were expecting, and ignore the others ( including disk full)
We fixed this by setting an internal flag when we detected this, and every half hour put out a message on the system log (managed by automation) saying disk full. (Without this half hour, the operators could have had 1000 messages a second).
When customers reported this message to our corporate help desk asking what the operators should do. The answer was "go and talk to the programmers".
The programmers were education on the use of an "else" statement.
Suddenly lots of erratic problems disappeared.
One of our testers said there are two extremes in testing
1) No matter what we did - we could not break it
2) If we were careful, and the wind was behind us, and we didnt push it - we could get all of our tests to run.
His favourite scenario was to limit the virtual storage allocated to each components and then hammer it. Including doing things in "the wrong order" and restarting it half way through. All that was "easy". The hard part was making it repeatable so that the same scenario could be retested with any fixes.
I was working with a customer for a week or two. While we were working some men were installing a partition to split the room into "machines" and "humans". We arrived on Monday to find the workmen had finished building the partition, but we could not get into the "machine" side because no one knew the combination to the secure door. Up came the floor tiles and someone (small) went under the door, refixed the floor tiles and let us all in.
For all of my time there, that door was locked open. I do not know if anyone knew the combination. I found out later that the workmen did not know the combination - they just installed the lock. At another customer I found that these locks had a default combination code - and most customers did not bother changing it!
I do worry that the wrong questions will be asked, the wrong answers being given and then people trying to fit their preconceived solution to the data.
eg
Q:What is the impact of replacing locally created solution with the corporate one? (Thus saving one person year of effort to support it)
A:We will need to develop a new process.
Result: It took 5PY to effort to develop and test a new process.
Success:! The 1py of effort decrease from the IT budget, (- lets not talk about the 5py of effort - it was "lost" in a different budget)
It said
" the volume of new or clarified requirements" for the system's HR module, which "emerged during the Build stage when business stakeholders were able to physically see the solution for the first time"
I thought these days ( this century) you gave people a working demo of the front end for people to use - before signed off the requirements.
I remember my sister (a nurse) gave lots of comments... for example.
"It is pretty that the boxes line up - but I do not need an input field 10 chars wide for Mr/Ms/M .. and a only 10 char field for surname"
"Why is the data I need about the patient on page 3 of the system and not the first page? I dont care about patient history when I'm trying to check the pills the patient needs"
"Where is the consultant's name?"
and so on.
The end user requirements had been "obtained" from the office staff - not the ward staff, because the ward staff were too busy.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
