Re: So many questions
If you are often taking the cover./backplate off, people don't screw it on - knowing they'll be in it next week. Touch it, and it comes off (whether you wanted it off or not)
283 posts • joined 23 Jul 2018
I remember being told about the connection between 2 mainframes - the cable was a maximum of 5 metres long. The usual configuration was ... 1 Metre from socket, down to the floor ... 3 m under the floor and 1 m up to the other machine. The customer's problem was the machines were 4 m apart, so it went across the room at waist height, over other kit and a chairback etc.
This worked fine till someone tripped and backed into the cable and pulled it out of the machines.
There was a bank of DASD - 7ft high . 3 ft square in slightly the wrong place (2 inches). They put their backs into it - and pushed with the legs... and moved the disks. As they stood up - someones's belt got caught on the Emergency Power Off button - which was pulled - and dropped the power. It took a day or so to recover!
Our team had 4 people in it, and it needed about 12, so some work was not being done - and our customers complained.
We were offered a senior person to help "understand the users and their problems". We had conversations like
Him: These users say they need a faster machine for their builds
Us: They don't need a faster machine; their current machine is lightly loaded all the time. The problem is the source is on a remote system the other side of the Atlantic. If they cached it locally then the build would be faster. They dont know how to do this.
Him:These customers complain because the network is wrong.
Us: They are trying to use the wrong configuration. They need to change their definitions - and think it will be quicker if we change the network rather than them fix their configuration.
We gave him some of the smaller problems to fix and he was very happy.
After a month he reported back... they need an extra 10 people to do the work. He almost wanted to join us because he learned so much.
A friend of mine who worked as a contractor in Europe was contemplating changing jobs. The company had had a bad year and were looking to cut expenses (and surplus fat).
The VP flew in on the corporate jet to tell them this news - the use of the corporate jet upset every one. The "surplus fat" included lowly paid people who did all of the useful little tasks, like tour the printers every day and check the toner and paper supplies, and "space planners" who organised office and site moves.
He said the last straw was having to organise his own office move, including several servers, all needing Ethernet connectivity (and so sockets to plug the cables in).
The day before his move he went to his new "office" and found the Ethernet sockets were not all wired in, and there was not enough power sockets to plug all the kit in.
He got back to his old office, and found someone had moved in!
(The VP did not just fly in to visit them - this visit was "squeezed in" to save another trip)
30 years ago I remember hearing about some software had to be installed - mandated by the country's government. It had close to 0 defects and very few fixes, and was held up as a good example of how software can be.
Until the government started doing audits to check it was installed, and insisted that the software be >activated< rather that just installed.
Activating the software killed performance, and the software had so many problems, the auditors agreed it could be turned off!
When the auditors checked that their own government's software was activated. they got told the same thing (no way are we running with this).
One of the sales people in the US for our product had a phone call to be at the corner of 1st and Washington at 1000 - that's all he was told.
A black Limo (black glass so he could not see out) picked him up and drove him around for half an hour, and then we went down, down down.
He got out the car, and was met by some marines with guns - and a receptionist. There was a flashing red light - which meant there is a visitor in the complex.
He was taken to an empty room (with mirrored glass at the far end), and voice over a speaker said hello, please give your presentation.
He gave his presentation to the empty room. At the end he was escorted out to the hospitality suite for coffee and doughnuts.
He wanted to use the wash room, and was escorted by a marine. The toilets had no doors, so he had to sit down and do his business in full view of the solder (with gun)
He went back to the presentation room and answered a few questions. After this he got back in the car and went up, up, up; drove around and was dropped off where he started from.
He does not know which US agency it was, but he said it was the scariest presentation he ever gave.
I remember visiting an all IBM, US customer, about 30 years ago. At lunch time the IT manager showed off his new car to every one (including us). At the same time the IBM tape drives were being removed and some other manufacturer's were being installed. I was told the two events were pure co-incidence.
We tried to use the tape drives that evening, and they didn't work. They called out the engineer to fix them and were told - he will be there within 3 days!
The on-site IBM engineer quietly opened the doors, checked the cables and switches and said "it is now working" - and quietly went back to his proper job.
This was mentioned at the end of week report to the management chain, and a few weeks later the IT manager had moved to a different company.
I think the "test sheets" were a like "unit tests" This section has 50 line per inch, this section has 60 lines per inch, etc. How about this green or this bluish green, or a bit more yellow in it.
From these tests they could decide on the best options to design the next generation bank notes.
Over time they could also see which way the technology was going.
My father used to teach in the Royal Navy. He would get a classroom full of people from the Commonwealth Navies and teach them. At the end of the first class, he asked if everyone understood. Most people nodded, and a few shook their heads. The people who nodded were allowed to go.
My father did the whole lesson again - much slower. At the end, he said "do you understand" and the class shook their heads.
Ever patient, my father prepared to do the course again, when one person said "We understand perfectly, when you ask use we all said yes".
In their country nodding means no - and shaking the head means yes!
As well as this little confusion they did not like to say no. The answer to a question "are you on target and ready to fire" was always yes. They did a lot of damage by hitting things outside of the target area.
Another problem was caused someone not using his right eye to line up the back sight and the fore sight, but using his left eye and only the fore sight, and shutting his right eye! This makes a big difference over a couple of miles.
40 years ago... I remember working with someone who was written a program which worked, They added a comment, and it failed to compile with internal error.
Take out a different comment it worked. Cue lots of head scratching from the rest of the team.
The problem was that for a program up to a certain size, the compiler kept it in memory. Over a certain size, it had to spill and use an intermediate file.
There was a bug in the code which spilled to the file. Obvious with hind sight
RAID is not always the answer. Disks will do what they are told, so if an operator says 'delete this file', or 'reformat this disk', the disk subsystem will do it. If it is configured for RAID, it will do it very reliably.
You still need backups.
At work, one of the printers had a paper jam. I cleared it, and it started printing its backlog. One of the documents looked like a list of porn sites. I took it to my manager who dealt with it. The offender was called in, and denied all knowledge of the document. The manager said he would call the IT department to scan the computer for this document, or any other suspicious documents. Suddenly the offender decided it might be better if he took early retirement.
I remember working on a bit world wide sports event where they provided email.
The email was scanned for bad stuff. Eventually this was scrapped as the scanners would not detect "We have your children -so lose" and the skater called "la bomb" got no mail.
Scanning images is more complex that this. The announcement needs to say how it will be scanned. If it is like Anti-Virus software will my phone get "updates" every couple of days as the parameters change?"
I did a health check at a customer's site, with a standard list of questions
Q:Do you backup? A:yes..
Q:How do you restore? A:We get John to do it.
Q:Where's John? A: He's on vacation this week... Oh he's turned his phone off.
Q:Can you show me the evidence that the backup/restore has worked?
A:Yes look. Oh Shot. We've been backing up and test-restoring the prototype database - not production!
They had moved from an evaluation system to production, and had not changed the backup jobs.
I added another question to my checklist
Q:Are you backing up the correct things?
In the the days when we had a tea lady who would bring your usual beverage etc. ( 40 years ago), we all had to work the weekend on a major project. The senior manager came round with the tea trolley and gave us all free tea's and coffees (and biscuits!).
He said if he asked us to come in - then he should come in. He thought the most useful thing he could do was to provide us with tea and coffee! He said it also gave him a couple of minutes to meet individuals in his department (some of which he had never met). I still remember him in his little pink apron.
We also had a senior manager who would add more paper to the printer.
Once I had a really urgent request to install some software. This involved a courier to got and get the tape etc.
I installed it over the weekend, and told the users it was available.
After a week I went round and said "do you want the password?" - cue lots of red faces.
I spent a couple of weeks at a customer working on an upgrade.
Our company had a rep attached to the customer who seemed to spend most of his time wandering around and chatting to people and not doing very much.
There was a problem with a database which had been going on for about a week. One of our company's database guys came on site, and quickly identified the problem, and spent the rest of the day doing a health check and answering database questions.
On the last day of my visit, I was having a beer with the rep who told me how he worked. In his wandering around he would hear about problems. He would then have a coffee with the manager of the area and say "I hear you have a problem with ... I can get a specialist in for a day to help at this ... cost" or "It sounds you are getting a bit short of CPU, I can get a salesperson in to discuss upgrades etc".
He would also quietly coordinate between different departments and say "do you think you could pop over to ... this afternoon, they have a question for you"
As a result there were few big problems, and the customer said overall it was a cost efficient solution. Our company got additional sales, and on site support contracts.
He did so well he was promoted to area manager, and hated every minute of it, because he was not working with customers.
One of my first jobs in IT (1980s) , was "application build". In those days DOS/VSE had one disk for the whole system!. Development tested on one system, I built on another system, and when the build was successful, switch disk, and re-ipl. So 2 disks in total.
I followed the instructions "delete the libraries using the ... command". 5 minutes later my manager came round saying the live system had gone AWOL.
I showed him the instructions and he said "ahhh. It should say use the build disk - see the white board, then delete the libraries".
I updated the doc.
Next week we had the same problem. My boss came in and said "you've messed it up again! Didnt you learn?" I showed him what I had done, and he said
"Ahhh, it should say update the white board to swap the disks round, then.."
Come the third week, I was in charge of a less critical build system.
I remember we had a graduate doing a year with us. He was outstanding - but very quiet.
After his year with us we all thought our company should hire him - and he would go far. A few weeks later my manager called me in and said the graduate didn't get through the HR interview, did we still want him - I said yes!
My manager escalated HR who came back and said "he was very quiet", and didn't have the social skills they thought every one should have. We replied that Einstein was a bit quiet - eventually HR approved him.
Some one in a different company said that HR pre vetted candidates and wanted people who ticked all of the boxes in their template. This meant that all of the technically brilliant people who may lack some of the social skills were excluded, and so they got technically average people (which my friend turned down as not being good enough)
I remember being at a company with a clean desk policy.
One guy would store all of his "in progress work" in the bin under the desk when he went home, and get it out next day. Which worked fine as the cleaners came round mid morning. "Security" would check the desks, but not the bins.
This worked fine until he had an urgent medical appointment one morning, and his bin was emptied.
He wasn't too upset - he said the design was rubbish and the module needed to be redesigned anyway.
A friend's mother had a laptop. When she changed her password - she wrote it down in a file on her computer. My friend had given himself a userid on the machine, and managed to find the file, and retrieve the password. His mother loved crossword puzzles. The file was called something like Kyber.txt Because everyone has heard of the Kyber-Pass Word.
I did a review of a system, and one of the charts had the line
"You are a very lucky company".
Initially the management very chuffed till I explained what it meant.
You have been skating on thin ice, and not fallen through so far. You had made changes without understanding the impact. If there had been a problem, you could not have backed out the change. As I said - you are very lucky you have had no problems.
In the days of 3340's which you could physically pickup and mount/unmount (and looked a bit like the starship enterprise), spinning disks etc.
One of our testers who was an operator in a previous job, had had problems with the disk containing the master database for the banks customers. He called over the senior operator who said.... we had better try it on a different drive in case the drive is suspect.
It didn't work there either - so it must be the disk. The got out the mother disk. Yesterday's database is copied to a different disk and the batch update run to make today's database (so Mother database begats today's database).
That didnt work either, so the senior op got out the Grandmother disk from the manager's cupboard. Mounted it - and it didnt work either.
So they phoned the manager who said "that's ok - just do not touch the grandmother disk".... "Ahhh too late - came the response".
There had been a head crash on the original disk.
Mounting it on a different disk drive damaged the heads of the second disk drive.
The mother disk was corrupted by the damaged heads.
The grandmother disk was then damaged by the damaged heads.
Fortunately they had a copy of the database which was only a month old, and could reapply the overnight changes which took about a week to do.
And that's when the tested decided to join our company where he could do less damage.
My father was in the Royal Navy, and they had some kit which didn't work. He suspected the suppliers were going to pass it on to some unsuspecting person. He had a junior rating scratch "condemed" inside the machine, and left a phone number.
A month later he got a phone call saying "we've just bought a new .... and when we opened it up, it had "condemed" - and your phone number.
I spent two weeks out in Asia as part of a team who were fixing one or two major problems at a bank. We solved lots of problems. At the end of the last Friday afternoon we presented to management about the problems we had found, and how we fixed it.
We finished the presentation and were heading towards the bar for a few beers, when someone timidly put their hand up, and said "but you haven't actually fixed the problem we asked you out here to solve". Coats off... laptops out.
It was a "simple" configuration problem. 10 PM that night they put the fix into production. It worked. Back to hotel at midnight, bed and depart 0600 next morning... so no beers.
From this I learned that it is worth walking round the techies at the beginning and getting their view of the problems. A common comment was "We've told management what the problem is, but they don't believe us"
We were working in an old building, and someone wanted to hang a few pictures on the wall,so had a hammer and some 4 inch nails.
One of the old hands told us to be careful. He said that the high voltage cable wandered around. It should go up and down, and be in a protective tube - but not in this building. Sure enough, we pulled the panel back from the wall to have a peek, and there was a cable just wandering around (in coils), not secured etc. It was not done by electricians, but by some people who wanted a "quick fix" until it was done properly.
The old hand said don't look under the raised floor - so we did. If people wanted to add new connections - they just added new ones - and left the old ones behind. It was 9 inches deep in cables.
I remember one guy who thought he was top notch, but was all talk. He would "work" long hours to get the job done (and claim the overtime) and told every one how hard he worked (He spent more time talking than working).
At one status meeting he was said he was half way though a piece of work, and it would take him only another 6 weeks to finish it. I spoke to his manager saying "You can either do this all by hand, or a write script to do the heavy lifting" I wrote a script in about a day.
The manager had a conversation with the employee. The employee decided that retirement might be a good option. It was that or be exposed at a status meeting.
After he left every one became more productive, because they didnt have to listen to him!
I was on a week's team building exercise. One morning we were split into teams and asked to discuss a topic and present on it.
Four people including a guy called John were given the title "working as teams".
When this lot came to present, there were two presentations. One from 3 people, and one from John both called "Working as teams". John could not see the irony of the situation.
Two similar incidents
A Canadian bank had two data centers - one on the east coast and one on the west coast. They wanted to use mirrored disks! The only problem was the network latency, so they moved the data centers closer to the middle of Canada, far enough apart to provide isolation.
(I heard of a bank in the US which had two data centers, one in North California, one in South California. Which was fine till someone pointed out the San Andreas fault line went through both data centers. They moved one east)
The other incident was someone complaining about performance of stuff across Asia. They said when they tested it - it easily out performed the requirements. When they rolled it out - the performance was terrible. I was one of a team who was sent out to help. In their testing they had two sites, but "one site" was on the second floor, and "the other site" was on the floor above. The network distance was about 30 ft! (Not the 4000 miles true distance between sites).
When we probed the requirements, they did not actually require each transaction to go to the remote site and back in under 10 ms. They could "batch up" the work. Do 1000 transactions, then check the status of the first transaction etc. Problem solved... every one very happy - especially as the CPU cost was reduced significantly.
I was on site where other people were trying to get a connection going. The remote end had configured the long random string password, and raised a ticket, so our end could set the password on our end.
They guys cut and pasted it, and it didn't work. Eventually they said, let's type it in. So one guy read it out, the other guy typed it. When they got to "O" the typist said is that an Oh or a zero? They picked Oh and it didn't work - they repeated it with zero an it worked.
The ticket raiser has typed the wrong password in.
From this I learned that every change should be cut and paste, and not typed.
a) It saves time (you do not have to think)
b) You can test it before doing it in production and be sure that what you are doing has been tested.
c) You have an audit trail.
I liked the version which mis quoted this.
In the grammar test James, while John had X had had Y, Y had a better result on the teacher.
Now replace X with <had had "had", had had "had had", "had had" had had >
and Y with <had had "had had", had had "had", "had had" had had>
You now get about 30 had's in a row.
It's turtles all the way down.
In the 1980's I heard the story about someone who wanted to get some more big (6ft high 4 ft square) physical mainframe disks into the machine room, but there was not quite enough space on the floor.
If they moved those disks 6 inches they could get the new ones in. So while the disks were in use (!) they leant back against the disk, and pushed ( with their legs as they had been taught) but could not budge it.
Unfortunately, one of the guy's trouser belt caught in the Emergency Power off. So when he stood up, the EPO was pullled and the whole bank of DASD lost power.
This took down the whole banking system - whoops.
Someone said that the best way to get the right things was to get the bosses to have to use it.
So I suggest the top brass responsible for this stuff get dropped into the middle of Australia or Africa, or Siberia and have to use it for a week in full battle kit.
How heavy are the batteries? Is there any WIFI? - No WIFI because of the terrain is hilly and there are dead spots.
I remember having to use a face mask in a hot dusty environment - one problem was the sweat would collect and slosh around inside it. Periodically you had to take it off, and pour out the sweat.
How is this different from all of the other frameworks the government has issued?
Have they looked at how other countries have done it, so they start from a working system?
Have they factored in all of the "lessons learned" from previous failures?
Have the pigs been fed and are they ready to fly?
Ive just experienced a similar but different problem. We were in a drama competition, where groups put on a play, and an adjudicator gives marks for presentation, acting, technical etc.
We had a play and a critical part was projection on to the back of the stage. The projection worked the day before.
When we tried it with out play, we found someone had hung a black cloth in the auditorium - blocking the projection.
Our lighting man got his own projector, plugged it in to the mains, and computer and it would display the set up screen, but not talk to the computer. He tried everything. He finally used a different extension lead thinking it will make no difference, and it suddenly worked.
The problem was that the computer and projector needed about 5 second to sync up. The extension lead had a dicky plug and would drop power for a millisecond, every second or so. As a result, it never synced up
This took him 60 minutes to find - all the time we were allocated to set up the technical stuff.
I had to visit a customer. The technical team were great. They had an idiot for a manager. He had been an excellent technician responsible for cabling.
The offices had glass walls, (some were opaque). The boss did not realise that the wall behind him acted as a mirror, and we could see that he spent a lot of time playing Ma-jong on his computer.
We presented to him, (and his peers), and his only comment was to ask us to change the header page, as the colours were wrong, and there was a typo.
This was a classic example of The Peter principle (https://en.wikipedia.org/wiki/Peter_principle)
"which observes that people in a hierarchy tend to rise to "a level of respective incompetence": employees are promoted based on their success in previous jobs until they reach a level at which they are no longer competent, as skills in one job do not necessarily translate to another.["
IBM's mainframes (VS1, MVS, VSE, the original DOS, zVM) running on 360/370/390 hardware were not written in C. 40 years ago they were written in Assembler, and PL/S (A pl/1 syntax like language). This evolved into PLX which is used these days.
These days application type stuff like web servers (and TCP/IP) may be written in C and Java. Even if you wrote in "Metal C" which was closer to the hardware, you still have to drop into assembler to do the hard stuff.
C has a rich library of functions like printf(), but the assembler and PLX had equivalents but they were not so easy to use.
I recognise these words. They are usually used during basic functionality, running a single task.
It they had said, "When running this at expected peak workload + 50%, with x thousand concurrent users" it performed well. I would be more impressed.
The problem with large systems come down to concurrency and locking/latching
Biting the hand that feeds IT © 1998–2022