* Posts by ColinPa

79 posts • joined 23 Jul 2018

Page:

You had one job... Just two lines of code, and now the customer's Inventory Master File has bitten the biscuit

ColinPa

Adding a comment sometimes caused compile failure

This is going back 30+ years when 1MB was a large machine.

Someone added a comment to some code, compiled it successfully, and submitted it to be integrated. This compile failed with a syntax error. Because it "broke the build" there was an investigation.

The team leader spoke to the junior programmer and stressed the need to compile before integrating. "I did compile it" said the trog*. Team leader said "ok we'll compile it again and show it fails" "oh it works - hey Charlie... here's an interesting problem..." and so on up the chain"

It turns out that when the original program was compiled, the source would fit in memory. When the comment was added, it was too large for memory, and so used "the spill file". This had a bug where it missed a byte when reading from the spill file - and put a garbage byte in the line.

* trog - troglodyte: in pre-historic times, someone who lived in a cave.

A tale of mainframes and students being too clever by far

ColinPa

Clearing storage may not be good for you

I remember a newish programmer working on a mainframe subsystem, getting a lot of storage (1 GB) and being a good programmer decided to clear it all using one instruction. This was very expensive, as it meant the OS had to allocate the pages on the paging data sets etc. Then because we were constrained for RAM the OS had to page in (and out) all of the storage as it was used. The performance people looked into why start-up was so slow and found this clearing was causing a problem

They found

1) It only actually used about 10KB of storage - not 1GB

2) If you allocate on a page boundary, the storage is cleared for you by the OS - so making the clearing unnecessary.

The "best practices" taught at college were not always the best, more like guidance.

'I'm telling you, I haven't got an iPad!' – Sent from my iPad

ColinPa

my data's gone missing

A customer phoned to say that our product kept deleting key data - why was it so bad? This was the first complaint we had heard so it was a surprise to us.

After a few days of to-ing an fro-ing they sent us the logs, they were getting more irate.

We went back and said

At 0430 on Monday, a userid JOEBLOGS issued a purge command ... Could this be relevant?

Whoops - thank you - we'll sort him out. They took the evidence to him, and of course he denied it, saying he was away. They brought in the CCTV and showed him sitting at a terminal typing at the date and time the event happened.

He decided to pursue opportunities in other areas.

Are you sitting comfortably? Then we'll begin. Hang on, the PDP 11/70 has dropped offline

ColinPa

psst goes my vacuum

Someone who was developing products that had to be tested in a vacuum (well low pressure), kept finding the was a blip in measurements during the evenings. They solved it by having "an evening in with Pizza" to watch what happened. The cleaner came along, are carefully plugged into a free socket and turned it on. The problem was that short cleaner had to reach over "the bell jar" containing the experiment, and touched it. This was enough to let in a small amount of air. The problem was easy to solve - just move the bell jar along the bench.

What evil lurks within the data centre, and why is it DDoS-ing the ever-loving pants off us?

ColinPa

Football team web site throttles business

I heard of a company where they found 10% of the traffic was from the local football club's web site. The football web site was sending traffic to web browsers every few seconds/minutes(I forget), even though the browser was minimised, or hidden behind real work. A quiet word with the club, and they fixed it so it only sent data when the window was active

£40m wasna enough for ink and toner cartridges in public sector, says Scottish government

ColinPa

How much for a nut

Reminds me of my father in the navy 40 years ago. He went to the ships stores to get a nut and bolt.. He was given a box of 144 of them. He complained - but was told that's the new way of working. He used the nut and bolt and left the box with the other 143 on the shelf in the engine room ... I expect it is still there.

When a deleted primary device file only takes 20 mins out of your maintenance window, but a whole year off your lifespan

ColinPa

Read and understand the instructions first

I got called out to help a "production down" problem during an upgrade. I was trying to help over the phone, and not being able to see what was going on was a major problem.

The instructions were clear.

1 Delete the following files config1,config2 etc

2 Recreate the system

3 Enter the config data when asked.

What could go wrong?

I got called at step 3. "Where is the config info we have to enter?"

"It is in config1"

"You mean the file we just deleted?"

They could not recover the file from the backups - because they were not authorised.

We eventually found the data because someone has copied all of the config files into one place for education.

We changed the instructions from "delete..." to "rename... " and added "step 0 - print out...".

Cool IT support drones never look at explosions: Time to resolution for misbehaving mouse? Three seconds

ColinPa

Re: Switching on the "monitor stand"

Comes from too many movies where the "computer blows up" and the display produces a shower of sparks.

Faxing hell: The cops say they would very much like us to stop calling them all the time

ColinPa

Whoops - hello Mum

Going back 40 years when you could use the office phone for limited personal calls, we used to get a piece of paper monthly with all of the personal calls made from the office. You work out with your office mate whose calls were whose, and pay the money into an honesty box.

One number was being called about 20 times a week, for about 2 minutes a time - even when I was away from the office. My colleague, Martin, denied it was him. so we phoned the number. The conversation went as follows "Hello xxxx building society", "Er hello mum, it's Martin, sorry, wrong number"

PC printer problems and enraged execs: When the answer to 'Hand over that floppy disk' is 'No'

ColinPa

Sending a child to do an adults work

I was visiting a customer in India about various problems - architectural and implementation. I was approached by a nervous young Indian girl about a problem they had ( same product, different problem). She said they had had the problem for a week, and not been able to solve it.

The documentation said if you get this problem - check the following a).. b).. c)

me:"Have you tried a)?" her:"Its not that"

me:"How do you know" her:"My team leader said so...."

by now the team leader and his manager (and lots of people had come round to watch).

me:"well could you try it?"

now her team leader stepped in

him:"Ive been working in this area for 5 years... it is not a)"

me:"Ive been working in this area for 20 years, I sit next to the guy who wrote the code please try a) we've spent more time discussing it than doing it"

her:"Well lets try a) at least... ahhh it has fixed the problem - thank you"

It turns out that the team leader thought he knew every thing, had to make every decision, and so make himself very powerful. He asked the most junior girl to come and see me, as he did not want to be seen not knowing everything.

The team leader went off in a huff. I spent more time with the girl who actually knew her stuff -she just wasn't very confident in her abilities.

The manager asked me to have a "quick look at the architecture", "hmmm I said, this wont scale, and if there is a power cut you'll lose this, and what happens when you need to reboot this box...." I think he was glad when I left.

'One rule for me, another for them' is all well and good until it sinks the entire company's ability to receive emails

ColinPa

Good riddance

We had a developer who spent all of his time talking and doing not much work. Customer's loved him because he could talk the talk. The rest of use muttered in our beards and did the work he should have done. He was "poached" to work for one of our customers as the architect. It was a classic win-win situation.

A year later at a conference I was talking to someone from that customer and asking how the new architect was. The reply was "He was bloody useless, didnt know anything, and was moved to a sales position within 3 months".

80-characters-per-line limits should be terminal, says Linux kernel chief Linus Torvalds

ColinPa

Re: not the terminal, the punch card

You quickly learned to get a marker pen, and put a coloured diagnonal line across the deck , from bottom left to top right. If any cards were out of order, it was immediately visible.

So you really didn't touch the settings at all, huh? Well, this print-out from my secret backup says otherwise

ColinPa

Re: Ah, customers.

I went to a customer who had lost the source of their critical application and it had a bug. I was asked if we could change the application server to get round it!

Not going Huawei just yet: UK ministers reportedly rethinking pledge to kick Chinese firm out of telco networks by 2023

ColinPa

Pot, Kettle, Black

Going back 10 years, I heard from a Chinese colleague, that the Chinese were worried about back doors in any kit they got from outside China. Hence their policy of "buy Chinese". They also said that a foreign country was caught trying to install a backdoor into Chinese kit - but was caught.

Das reboot: That's the only thing to do when the screenshot, er, freezes

ColinPa

Check your background

I recently saw a document which some marketing guy had developed, and was about to publish to the world. It was meant to look really modern. The diagram had a white background, and had off-white lines linking boxes. The diagram made no sense. If you tilted your screen a certain way you could faintly see the lines. When I changed the background to a different background, it was obvious (and still made no sense).

I also asked what the thatched cottage and museum icons meant, and suggested they have hover text to explain what the items were, but have not heard back from him.

If American tech is used to design or make that chip, you better not ship it to Huawei, warns Uncle Sam

ColinPa

Re: Also Microsoft

How would you stop them? If I was the Chinese, I would use the software without a license, and block the IP addresses that Windows uses to phone home.

You overstepped and infringed British sovereignty, Court of Appeal tells US in software companies' copyright battle

ColinPa

Re: Watch Out USA...

It wont work - you wont be able to take the halberd into the US. It is a lethal weapon.

Mirror mirror on the wall, why will my mouse not work at all?

ColinPa

problems getting started

Going back about 15 years we got a laptop for my aged mother. We set it up for her, draped the phone lead from the phone socket in the hall, across the room to the table, connected the power from the kitchen to the eating area and got it working.

I didnt see her for a month, but when I phoned she said she had problems getting started, the lid would not open.

Next time I came to visit,I got her to show me the problem. The lid would not open, because she was trying to open the hinges. She had put the laptop down so the logo faced here (obviously). (The logo is to show everyone else - not for the user) .She knew she had to release the catches to open the lid, and the hinges had a sticky out bit.

Putting a label on the front "open here" was a big help for a while.

Remember - "Things are only obvious once you have seen it".

'We're changing shift, and no one can log on!' It was at this moment our hero knew server-lugging chap had screwed up

ColinPa

I just cleaned up all these unused data sets...

On the mainframe, we had a new, keen data manager join my team. He had just been promoted from assistant, to full data manager. After a week of looking at the environment, and listing what was used etc. he got to work, and deleted lots of files which had not been used in the last 5 years. He managed to recover about half of the disk space so was very pleased with his work - till Monday. On Monday lots of people complained because about 1000 CICS regions did not start, because lots of data sets had been deleted. The existence of the file was checked at startup - they were only used when there was a problem.

Fortunately these were test systems, but it took several days to recreate the files.

Two things he learned from this was to check before making changes - and to set up rules to categorise data sets into "backup" or not-backup.

Behold: The ghastly, preening, lesser-spotted Incredible Bullsh*tting Customer

ColinPa

"Lying" to your boss

We had an Asian customer who rolled out an application and very publicly announced it. The problem was that some bank transactions done last thing at night got lost. After a day or two there were tweets like " do not do your online banking before you go to bed if you want it to work". A major problem to the image of the bank.

The manager got on to technical support and wanted action IMMEDIATELY. Of course he could not wait for us to look at the problem, and he escalated it up our management chain. He went to his boss and said as it was all the vendor's fault, the vendor( us) was going to fix it. It was all about loss of face etc. From our side it took a lot of effort to handle the management flack - more than the effort to "fix the problem".

The memo back to the customer - copying all the important people, was very calm and along the lines of "We have looked into this problem and there is no product defect. We can see that the records are deleted because at 0030 each day a userid AUTOOPS1 is issuing the command to clear the messages on the work queue. This command was successful in deleting the requested messages". It turned out that in testing, they ran an automated job overnight to clear old messages from the queue, to get a nice clean system for the next day's testing. This code had been ported into production, so any messages submitted just before this job ran were purged.

I heard the next meeting was a blood bath - The CTO said he had phoned the CEO of my company and asked for help, and it turns out to be a simple user error! A major loss of face for the CTO. We never heard from the manager again and no one would tell us what happened to him.

Britain has no idea how close it came to ATMs flooding the streets with free money thanks to some crap code, 1970s style

ColinPa

Approaches to testing

We had one tester who had some sayings

- There are two ways of reporting testing. 1) "We have run all of the tests, and on a good day with the wind behind us, they all passed" or 2) "No matter what we did - we could not break it". Management want 2) but will take 1) and run.

- The point of test is to change the product until the all the tests pass.

- Automation is great - but eventually the product is fixed and all the test run. You then change the automation - slow it down, speed it up, test different things.

- If you haven't broken it - you havent pushed it hard enough

My experience of catastrophe prevented, was testing on the biggest z/OS mainframes configured for High Availability, running CICS, IMS, and MQ. I ramped up the transaction rate and within minutes one mainframe had crashed. But this was configured for HA... so the work seamlessly moved to the next mainframe -for a minute, and this then crashed - and the work moved to the next mainframe - which also died.

No one else had this problem not event the stress team - I thought it was my dodgy monitoring code.

It turned out that under extreme load some code was not freeing "system storage" because someone had copied only 99% of a fix from somewhere else. They added the one line of code saying "this block is free" and it worked, and I could not get it to fail.

Academics demand answers from NHS over potential data timebomb ticking inside new UK contact-tracing app

ColinPa

How to stop people from having "fun"

What protection is there to stop some joker having fun and broadcasting "Ive got it" to every one, and leaving phones hidden outside a hospital.

It would be good to read the plans to prevent this sort of thing.

UK snubs Apple-Google coronavirus app API, insists on British control of data, promises to protect privacy

ColinPa

if I was malicious....

There are always people who want "have fun" either maliciously, or because it is a challenge.

If I wanted to be subversive how easy would it be for me to produce false data?

Do we need to factor this in to the high level requirements and design of the end to end solution, and see how this drives the implementation?

Could I have a phone outside a supermarket capturing data of people walking past, and then broadcast "I have the lurgy", and sit back and watch?

Australia's contact-tracing app regulation avoids 'woolly' principles in comparable cyber-laws, say lawyers

ColinPa

Does this use bluetooth?

How does it get info from nearby contacts?

By default my bluetooth is disabled as it drains my battery.

How long is "a contact", You are close to someone for 30 seconds and they sneeze. Will the app pickup this contact?

A paper clip, a spool of phone wire and a recalcitrant RS-232 line: Going MacGyver in the wonderful world of hotel IT

ColinPa

Danger - building works

In the days when the heat from the water cooled main frame heated the site and before PCs, we had a new building build next door. One day the machine digging the foundations put it through the main water supply for the site (I heard the effect was like a whale blowing). This meant no main frame, no air conditioning. As this was a "modern building" this also meant there were no opening windows ( or outside awareness panels as they were called). It took a week to fix, and the boyd odour smell was not pleasant.

In the last 10 years I visited a company who had taken High Availability very seriously, including dual network providers etc. The both came in the front of the building - in through the same hole in the wall. This didn't help when someone did some work in the road and cut though both network cables.

I also heard of the case when a French company had two network suppliers to cross France. One cable headed north, one cable headed south - what could go wrong? What happened was both cables were rerouted to use the common network infrastructure across France. At the other end, one cable went north, and down to the site, and the other went south, and then up to the site. Im sure youve guessed what happened. Something happened to the common network infrastructure and so both networks were broken.

Vodafone chief speaks out after 5G conspiracy nuts torch phone mast serving Nightingale Hospital in Brum

ColinPa

if you dont like 5G - dont use it.

As a lot of infrastructure will use 5G, could we just have a FB button which people press which says "I dont want to use 5G". This sends details to their supplier who stops their phone from working "over 5G".

Along the idea of "I dont think motorways are a good idea they cause pollution. Why is there no food in the shops?"

COBOL-coding volunteers sought as slammed mainframes slow New Jersey's coronavirus response

ColinPa

This is 2 additional transactions a second

206,000 a week - 5 days a week 8 hours a day is 1.4 new requests a second.

What are they doing in their transactions You can do thousands of credit card transactions a second

ColinPa

Solving the wrong problem...

I suspect they do not need cobol programmers.

1)They need application programmers if they need to fix application bugs

2)They need to contact IBM and borrow a faster box. If they have space in the machine room, wheel it in, plug it in and restart. Old systems should still run on the new box. Neweer boxes have even more RAM.

3)The box may be OK, but the database is not. A faster box may help, but not if they are having deadlocks or the applications have built in serializations. These days ever things should be in processors memory or disk cache

4)There is an application bottleneck for problems like "hold this lock while I chat to the end user" a faster box will not help.

I suspect techies know the problems - management do not understand.

That awful moment when what you thought was a number 1 turned out to be a number 2

ColinPa

Re: How to write directions...

We got some new graduates to install our software to connect this box and that box and produce a report. We gave them exactly what we give to our customers and the results were embarrassing.

- They did not know there were installation instructions on the web. They just googled the product and followed someone's blog post

- They did not know anything about TLS ( SSL) - they tried things at random until it worked. They did not know there was an error log reporting problems

We made changes to help them get started. We tried it again with a new set of graduates - and they totally ignored our instructions again!

ColinPa

Re: Technical management tips

The best manager I had had some rules for a crisis.

1) Leave the technical people to it

2) Do not stand over them because that will make it worse

3) The manager would come in every half hour for a status update - bringing in tea/coffee biscuits. It gives the technical folk a break from the work face, and make sure they were working on the right problem by talking it through.

4) If the technical people want something - the manager organises it (pickup kids from school was one example)

5) The manager takes all of the calls from irate executives etc

Dell publishes data centre cleaning guidance, suggests hiring pros to disinfect enterprise kit

ColinPa

For the bits inside that isopropyl alcohol does not reach

Should I do a defrag?

I heard somebody say: Burn baby, burn – server inferno!

ColinPa

Re: Oh so special's

40 years ago we had a lecturer whose specialty was 10 dimensional space. He was considered "odd". He was always muttering to himself, and he rode a bicycle because he didn't drive. In his view it was easier to be in 10 dimensional space than the real world.

He is a different lecturer from the one who put his pipe in is jacket pocket, and the jacket caught fire.

Researchers trick Tesla into massively breaking the speed limit by sticking a 2-inch piece of electrical tape on a sign

ColinPa

If a piece of tape can do it - what about dirt?

Some of the road signs get covered in dirt - or even speckled with dirt ( think gritted road, cars going too fast), spider crawling over the sign.

What problems will this cause?

C'mon SPARCky, it's just an admin utility update. What could possibly go wrong?

ColinPa

Just following instructions....

I find it amazing that installation instructions (from software companies that should know better) start with "go into su mode.... and issue the following instruction". The developers have their own sandbox machine, ("just spin up a cloud instance for the test so if you break it, it doesnt matter") and have clearly never been near a production environment.

Who needs the A-Team or MacGyver when there's a techie with an SCSI cable?

ColinPa

Lonely machine needing company before it would work

I was told this in the bar at a conference, so cannot vouch for its 100% truth....

In the corner of a big machine room were a couple of machines. They moved one (as part of an upgrade) and the other machine stopped working. They moved it back again and the other one continued to work, so they left them both there. They found out a few years later that there was a maze of cables under the raised floor, and one cable was joined to another ( and stretched tightly). When there was no weight on the raised floor the the contact was broken. When there was weight on the the floor tile the cable relaxed 1 mm and the connection was made and the machine worked.

Tech can endure the most inhospitable environments: Space, underwater, down t'pit... even hairdressers

ColinPa

Dumb terminals - dumb users

In the days before PCs our company wanted glossy colour photo of the test team in action, for the external magazine. Cue picture of 10 people at 10 green screens, all showing the logon screen. Because the test team had worked the weekend, they did not come in on the Monday, so they rounded up 10 people to pose at the dumb screens. This included a couple of secretaries, the cleaner, a man who had come to fix some electrics, and some managers. They all posed hands in the air like meerkats, for the photo. The photo got a commendation for showing lots of women "working in IT", before someone pointed out the logon screens and that none of them "worked in IT".

As a result, someone took a picture of the real test team, complete with a bin full of coffee cups, full ashtrays, the pizza boxes on the floor, what looked like someone bashing his head on the screen.

So you locked your backups away for years, huh? Allow me to introduce my colleagues, Brute, Force and Ignorance

ColinPa

Old rules from the royal navy (100 years ago)

- If it moves, salute it

- If it doesn't move paint it

- If it doesn't work, hit it with a hammer

- If that doesn't work, get a bigger hammer

Clunk, whirr, buzz, whine. Shared office space can be a riot and sounds like one too

ColinPa

Re: OpenOffice

Some people would have "ping - you have an instant message" enabled. I used to prowl around the open floor till I found the culprit. Often they would say "I didnt know it was on - ping - oh that noise". It was often hard to find the culprit because it was hard to locate, and sometimes the ping bounced off the walls.

Beware the Friday afternoon 'Could you just..?' from the muppet who wants to come between you and your beer

ColinPa

Im a brain surgeon - dont ask me to fix your car.

Go back to before windows XP days when I used a "green screen" in the office. I was a troubleshooter on the mainframe, and would be sent around the world to help sort out problems. Someone from up the road knocked on my door and said they had heard I worked in computers - could I help. They had a "new printer" - which I think had been nicked from a scrap heap. They plugged it in and it would not talk. I explained I had no knowledge of PCs and said they might need to get some drivers for it - did they have the diskettes? They glazed over at words like "driver". They didnt have the diskettes so I left them to it. The next week my neighbour told me that I was being bad mouthed up and down the street because of being "a computer expert" and knowing "nothing about computers". I said it was a bit like asking a brain surgeon to fix your car. The good thing was I wasnt asked to help for any other problems in the street.

The delights of on-site working – sun, sea and... WordPad wrangling?

ColinPa

FTP Mangler

One weekend when I was on site, the z/OS customer had a problem for which there was a fix. The usual people who down loaded fixes were not it, so I downloaded it, emailed it to the guy sitting next to me, and he uploaded it. It fixed the problem, but occasionally dumped. We decided not to put the fix into production.

When I looked at the dump, I could see the module was corrupted. It dawned on us that during the download and upload different character sets had been used, and so a few instructions were subtly different. We learned not to take short cuts.

The time that Sales braved the white hot heat of the data centre to save the day

ColinPa

The quiet hero almost never gets the beer.

I was visiting a small city in the US to upgrade some software about 30 years ago. We could not work on Friday because the "old" tape drives from IBM were being replaced with some nice shiny ones from another supplier. At 4pm the engineer departed saying "see you on Monday". At 5pm they started the weekly payroll run. This had to run or else the city employees did not get paid, and so they buses did not work etc and the city stopped.

At 17:05 there was a problem in that the "tapes would not load". At 17:30 they still were not working so they called the support number for the new tapes. "Sorry, they have officially not been installed yet. We'll send someone out on Monday". They phoned the IT manager who was not in "he's out driving his new car". Was there a shady deal, new tape drive -> new car?.

Meanwhile the IBM engineer who was doing paper work in his cave, quietly came out, took the covers off the competitors tape drives and fixed it in about 15 minutes. He came past our desk and just said, "they are working now".

On Monday there was a crate of beer hidden in the cave. No one could shout about the good deed the IBM hardware engineer had done, because it was not IBM kit.

Next time I came to visit there was a new IT manager.

It's always DNS, especially when you're on holiday with nothing but a phone on GPRS

ColinPa

No good deed goes unpunished

I was one of a team providing on site L3 support for the Olympics. We had nothing to do, so stood around for an hour or so watching the games. We moved to the coffee machine to get a drink when our phones went crazy. 100 missed calls, texts like "What is this problem you have", "please get John to answer his phone" etc. We had calls from techies all the way up to executives! No one had realized that there was a dead spot for mobiles in the room. We asked around for the problem, and no one knew of any problems. We got in the car and went to the venue.

When we got there, there were two windows machines one was fine the other was so busy it was hard to logon. Someone had configured antivirus to run every 10 minutes instead of once a day (at midnight). We stopped anti virus and the machine woke up.

We asked who they had called when they found the problem. The people had ignored the instructions - "any problems phone L1 support on xxxxxx". They had a list of phone numbers for all products on the machine and picked one at random - our product.

It took us several hours to reply to all of the emails and calm down the management chain. This incident had percolated up the chain to the CEOs office and they wanted action. Our management team were good - they managed to persuade the executives that we did not need a task force to determine the root cause. We were an innocent party - we were the good guys for fixing it - not for causing it.

Our senior manager told us "remember no good dead goes unpunished"

El Reg presents: Your one-step guide on where not to store electronic mail

ColinPa

Storing documents in a real bin

We used to have a clean desk policy and you could have no paper on the desk or walls. Rather than carefully put his papers into a drawer, and sometimes forget to lock the drawer, one guy would put them in the waste paper bin, and retrieve them in the morning. He said this was foolproof because the people who did the clean desk policy did just that - check the clean desk. They did not check the bin. The cleaners came after lunch when the papers would be on his desk. Genius

Remembering Y2K call-outs and the joy of the hourly contractor rate

ColinPa

Whoops whats happened to the coffee machine.

When working in a large multi national IT company, we had tested all of our products, and had a team on site over the new year. Besides the odd customer application problem there were no major problem - except for the coffee machines - they refused to vend. This was cured by someone one phoning home, and having their coffee machine brought in ( illegally because the power cord had not been checked by the on site electrician). Of course having had lots of caffeinated coffee's people could not sleep.

I flew to the US on Jan 1st to provide on site support for one of our worried customers. The biggest problem there was caused by people firing their guns (with bullets) into the air at midnight. Of course what goes up - must come down, and there was a spate of head wounds in the local hospital!

Wham, bam, thank you scram button: Now we have to go all MacGyver on the server room

ColinPa

Dont have your machine room at the top of a building

I was at a IT conference, and we were discussing horror stories. As it was in the bar, some of the details are hazy. One guy said that someone spotted smoke coming from a display, pressed the scram button, there was power down, and the halogen system was activated.

To reactivate the halogen system, they could not get the kit up in a lift, so they had to take out an exterior wall, and have a huge crane lift in some kit (Halogen tank?) to the 13 floor, rebuild the wall, and "clean" the machine room.

There was a 4 month outage all because some insulation on a screen was starting to smoke.

Not all details are accurate, but I hope you get the idea.

Remember the Dutch kid who stuck his finger in a dam to save the village? Here's the IT equivalent

ColinPa

This is one comman you never, ever type in - whoops.

I heard from one junior operator, that one of the senior operators was doing education to the newbie operators on the production system. The teacher said "this is the command to clear the spool, that you never, ever type in, or you will get sacked. CP PURGE RDR ALL SYSTEM".

and of course,he automatically pressed enter.

The class then had a master class on how to recover from a self induced disaster, as his phone rang about 1 minutes later, and he could not lie and get out of it.

The junior operator said it was a very good lesson on how to be careful, and manage a critical situation.

We've found it... the last shred of human decency in an IT director – all for a poxy Unix engineer

ColinPa

The senior manager wearing a mob cap and apron

This was 40 years ago when I had been with the large multinational for under 1 year. These were the days when we had a tea lady who had a tea trolley who brought you the morning gossip, a cuppa, and a biscuit.

We had all been asked to work on Saturday for the final push to get the product through the door. We were pleasantly surprised when the senior manager (responsible for 150 of us) came round with the tea trolley, wearing the statutory mob cap and floral apron to serve all tea and biscuits. He said that as he had asked us all to come in, he thought what could he do, so came round with the trolley. He went round and round the building and was one of the last people to leave. He said it was a great way of meeting the team.

Would it happen these days? - No we lost the tea trolleys 30 years ago.

That code that could never run? Well, guess what. Now Windows thinks it's Batman

ColinPa

"Error handling to be added"

I worked on a huge multi release software project. One of the line items in a release was a "5K lines of code" item for a developer. This line item expanded to 20K lines of code, the software was eventually released, and the guy got a big award for his major achievement.

The problems started when the support people got calls about it not working, or failing to recover after a problem.

The code had big comments in many place saying "error handling code to be added". The code only handled the golden path!

In the ensuing witch hunt, it turned out that as well as the lack of error handling, there was no serialization in the code. Testing had only had one or two transactions running at a time on a two core machine. Customers were running on multi core machines, and so there was a higher risk of hitting a serialization problem The code as written ran very fast, once the serialization and error handling were added, it was significantly slower.

The irony is that most of the code was never executed, as the specification of an interface was badly worded. We found this out when running all of the tests, and the coverage tools showed big areas of the code was not executed. The service people gradually took out unused code and no one noticed.

Hyphens of mass destruction: When a clumsy finger meant the end for hundreds of jobs

ColinPa

thin fingers, small brain

I remember working with someone on z/VM where they had carefully created synonyms: E for Edit, BR for Browse, and ER for erase.

So the sequence was BR to browse the file, it is the right file, retrieve the command and overtype E for edit - whoops the file has gone as the command was now ER ..

When I pointed this out to him he said he wondered why half his files kept going missing every day.

I helped him remove his ER synonym, and removed his ability to write to common disks.

Don't look too closely at what is seeping out of the big Dutch pipe

ColinPa

A side effect of fixing the printer jam

I used to do my bit to help the department by fixing paper jams and putting more paper in to the printer. After fixing one jam the printing restarted, and printed out a long list of hard core web sites. I took the list (with the cover sheet identifying the owner) to my manager. My manager handled it properly. Later he said that he spoke to the person who denied it was him who printed it out. My manager told the employee that we could give his laptop to the IT department to look at the browsing history. Suddenly the employee thought it was time to leave employment and take early retirement.

Page:

SUBSCRIBE TO OUR WEEKLY TECH NEWSLETTER

Biting the hand that feeds IT © 1998–2020