It's the day before the grand opening but we need a firmware update. It'll be fine Before one can organise a piss-up in a brewery, one must first get the brewery started. Something a Register reader found difficult in today's Who, Me? Our tale takes us back to the '80s, the decade of fun. Our brave reader – let's call him "Gareth" – was heading up a team building a state-of-the-art Brewery Process Block. It …
There's an unwritten rule in the IT world about Friday afternoon deployments?? Unwritten my ass! I wrote that in many memos!! It only took one overnighter on a Friday to figure that one out, no support after 5pm or on weekends! So happy to be out of full time IT now!
Beer icon has probably been used to death, but what the hell!
Depends on the system as to what time do IT Deployments
Our IT department have done deployments at 2am on Sunday mornings. With full regression testing during the day to allow for rollback in case things go wrong so that system is working on Monday.
When you have automated order and billing systems have to do when quietest. As have to put hold on through put
I agree but 9.999999/10 don’t touch anything on the last day before a weekend/holiday.
My last WAN migration for a government department was planned for a Friday after close of business. 3 months before the change I was told I could have the whole WAN off line from Friday 6pm through Monday 7am.
The week before the change I was told I couldn’t start before 8pm on Friday and multiple sites needed the WAN from Saturday 7am as loads of staff across multiple sites where doing overtime.
Due to good planning we where done all 30 odd WAN routers by midnight including 3 dc’s, everything tested and no dramas in the morning. Yes the new WAN was pre staged, connected & live, requiring cutting routes over with the benefit of a not too difficult rollback if needed.
Of course I was logged in at 6 to double check and no reported issues at all during the day.
They never normally go that smoothly, I got the sense that people where wondering what the fuss I was making about having a weekend of downtime was about.
A classic example of perceived crying wolf all because of meticulous planning, huge amount of pre preparation & faultless execution resulting in a positive outcome.
A classic example of perceived crying wolf all because of meticulous planning, huge amount of pre preparation & faultless execution resulting in a positive outcome.
Pretty much like the "millennium bug " then! I still come across the profoundly uninformed/stupid voicing their opinion that it was all a fuss over nothing.
If we are allowed to prepare we get "well that was a waste of (over)time then. If we are not, we get blamed. C'est la vie!*
*If said with a Scottish accent, that phrase is pronounced "' 's a lavvy" which to those from the remote south means "it is a toilet". Both the French and us are referring to the same life!
In the years leading up to 2000, I got paid an awful lot of money re-certifying stuff that had been certified to be Y2K compliant some 10-20 years earlier. Same for the embedded guys & gals. By the time 2000 came around, most of the hard work was close to a decade in the past ... the re-certification was pure management bullshit, so they could be seen as doing something ... anything! ... useful during the peak of the dot-bomb bubble.
Look for similar bullshit/misdirection during the end of the first UNIX epoch in 2038 ... despite the fact that all of the important systems that would be affected either already have been, or can easily be modified, making to so-called"problem" non-existent.
I agree, and not being an engineer I wouldn't dare not. But there's a caveat. A lot of perfectly good, non-system critical, non-networked devices were "millennium-proofed" at significant cumulative cost. Or simply mothballed/dumped. There was no rational risk-assessment performed on these devices.
PCs ( some quite elderly) used for word processing did not need any of that. What was the worst possible outcome? They'd fail to boot in January 2000. Improbable, but manageable. No different to when a cmos battery fails.
And I did take one or two out of mothballing a few years later - they were useful for taking off-site for kids to use etc. Worked perfectly of course. (Though the CMOS battery failed, in at least one - might have been coincidence).
I seem to remember some 'Y2K compliant' card scanners fell over a few days before Y2K because the merchant system added a couple of days to one of the fields to cover time to allow the transaction to clear but that field hadn't been tested properly. It had been tested for the Y2K rollover and would all work properly once Y2K had kicked in, but in the last few days of the century it either overflowed or was flagged as 100 years past
Then they had to go for a firmware upgrade. With no apparent good reason.
Here's an idea : you fully test that the system can do what you need it do the next day, then you make up your mind on the firmware upgrade.
If the test had did badly, then yes, you upgrade, but if the test went well, you don't touch a damn thing until after the event.
Oh well, experience is what you get when things break unexpectedly - especially when it's your fault.
I recall discovering that a major event was about to happen just after I started updating the system.
I discovered this when a security barrier and nice guard suddenly appeared, blocking my route back to the control room.
Fortunately he let me enter, and I was able to complete the updates and bring the system online half an hour before the Spice Girls came on to announce they were re-forming (again).
I don't know whether I could have saved the world from this fate by being a little slower or making a mistake, but sometimes I wonder...
I remember going to a presentation of a great new technology and just after the demo started Windows update kicked in. The download activity was enough to impact the network connection, and instead of a slick demo it was a stuttering display. The audience offered suggestions on how to kill the update, but the marketing person did not want to go off script.
The root cause for this is almost always one ting in my experience....
The presentation is usually carriend out on a machine who's ONLY job is in a meeting room.
A few years back, we would have a meeting every Thursday (you see where this is going) - the only meeting ever in that room @ 9am
.....and at 9:05am, it would dutifully start applying software patches pushed out to it by whatever patcher we employed.
It took a month before someone realised it was beneficial to move the meeting, but switch the kit on well in advance - or the day before to patch in advance
We generally have a good record on updates. That's probably something to do with the fact we use system center to restrict them to running between 5pm and 9am, and have it set to wake up any sleeping machines during the night to check for updates, as well as power up all machines at 8am.
We also have an "at risk" period from 7am to 9am on Tuesday, which we use to make changes to production servers (after testing, of course).
The system is far from perfect, and we do get caught out, but we rarely have a situation where someone is giving an important presentation and the machine restarts to install updates.
Back in the Windows XP days, my company had a standard image with a tool that would checksum c:\program[sic] files, c:\windows etc to make sure that all patches had been applied and nothing had been changed.
This wasn't a bad idea in principle, but it was configured to run every four hours and in the days of single core Pentium 4s and hard discs this meant your laptop turned into a paperweight. This actually killed a number of high-profile presentations and generally made it hard to use your computer
In the education industry, the interaction between options baked into standard operating system images, users with limited rights unable to change option settings, and power management hints to external monitors (e.g. screen projectors) causes a never ending stream of multilayered fun.
Add in OS updates and separate application auto-updates for bonus laughs.
Surprised a computer inside a surgery room would even have an internet connection. I know atleast one hospital in NL has the computers running the surgery equipment and such all airgapped with only a single network link to a single computer outside the OR. Only way to get data onto them is through that computer outside the OR (Which is ofcourse loaded to the gills with software to prevent any nasties getting in and with a very strict firewall to the cables going to the OR equipment. Also decreases the risk of contamination as data carriers don't need to be brought into the clean environment.
A few years ago a (semi?)pro basketball team had to forfeit a game due to windows update. IIRC, the laptop that ran the arena scoreboard had failed just before game time. They actually had a spare laptop on hand to run the scoreboard, but when they plugged it in, it insisted on performing a windows update.
Unfortunately, the backup was fairly far out of date with updates, and they ended up taking a very long time to complete.
Granted, modern scoreboards have a lot of fluff for entertainment value, but they also provide the basic scorekeeping and timekeeping functions that are mandated by the league. Per league rules, the home team was responsible for providing the scoreboard hardware, and if the hardware failed, they could only hold the start time for a limited time (40 minutes, I think) before the result was a forfeit.
After reading this post (and the rest in this thread), again I have to ask how many billion man-hours has Windows cost the Corporate world over the last 20 years? How many perfectly good dollars has that translated to? And it's getting worse from what I can see ... so why on Earth do the Corporate Lawyers allow the silly clusterfuck into the building at all?
More to the point, how much is Windows contributing to the global recession?
Answers on a postcard, please ...
Second only to Adobe Upgrades. Enabled by default and feel the need to pop up a bloody huge red box covering the screen. Designed to occur whenever a presentation is being made.
Which is why I refuse to use Acrobat Reader. First thing I removed on my corporate-issued laptop.
Every component of interlinked systems that I have every worked on (Including the human-driven ones!) always involves flow control somewhere. In-band or OOB, it's always present.
IMHO, flow control make thing work smoothly, and 'bottleneck' restrictions are easier to identify, as long as you bother to have flow on/off assertions visible somewhere. You can then tweak the setup as required to improve things.
Also, if one is dealing with something synchronous you need to 'Think Synch'...
Should really only be applied when.
1. It won’t run without them I.e. a serious bug
2 there is a massive security issue i.e. getting hacked
Otherwise test it properly before starting- but hindsight is wonderful…
Certainly not the night before a demonstration or high profile event.
If you need to do a fix then test it first, years ago I worked for a company building control machinery for factories, and one I remember if the factory got shut down as it made plastic sheeting it would have taken a week to get it back up and running, so that was approached very carefully. The problem was the heaters went off when the stop button was pressed so all the plastic solidified in the machine…
3) Before the system goes live, because you know that getting approval to update the firmware once it is in production is way harder.
You want the firmware as up to date as possible because when you call in for support on anything hardware related one of the first questions from the support engineer will be what firmware you are running. No matter what hardware issue you have, if the SE is able to dig up something in the bugfixes for firmware updates newer than yours that might conceivably impact it he will try to insist that needs to be done before treating it as an actual hardware issue. Even where you get them to table that and look at other things, they will keep coming back to that with greater and greater vigor the more things you rule out.
The closer to the latest firmware you are the less chance he can finger a potential firmware fix as the solution to your problem.
The mistake "Gareth" made was waiting until the day before the grand opening to do this. He should have done it right before they started their QA / acceptance testing, because you want as few (ideally zero) changes to the system from the time you start QA to the time you go live.
When we get the service engineers in to fit replacement parts to the robots, I have to pin one of them against a wall by the throat and tell them not to install updates to the OS/software as I've no wish to play "why the f does'nt this proven programming work any more?" for 4 hrs before finding out the system was updated the day before...
Update when the cell has'nt got 10 000 parts to make for delivery yesterday is the rule....
The Internet was quite common in business in the mid 1980s. I have business card binders (remember them?) from that era, nearly all of the cards contained in 'em have email addresses ... mostly joe@example.com addresses, with a few vestigial bangpaths. All of the latter were either Uni addresses, or research facilities attached to Unis and most of these also have a second @ address.
The article brings up memories from dfecades ago when I was designing system based on 68HC11. One chap in the team was not too impressed with BUFFALO and rewrote his own spiffy monitor. Loved working with it tho, must dig through boxes, must still have some cards left somewhere.
The cycle of UV-erasing was fun especially if you didn't pay attention to slight variations of how long was needed for erasure or what the programming voltage of that specific EPROM needed to be.
> I can remember cooking EPROMs at 200C on a baking tray in an oven for half an hour
Now I know what I was served the other day! I knew it couldn't be pastry, even if it was apparently still warm out of the oven.
(I just hope the machinery they installed the actual coffee cakes in wasn't mission-critical.)
At least it all went well in the end. On a smaller scale, years ago I and a colleague were to install a piece of software written by one of our junior developers on a remote system. Our main task was to inspect and service the hardware, but while we were there we would install this new gui tool as a freebie.
The night before in the office, the developer was doing some unnecessary tweaks to the software, we insisted they change nothing to the version we already had on the 3.5" floppy, ready to pick up early in the morning.
The next day, the hardware service went fine but when we went to install the new software ... it didn't work. Lots of head scratching and calls back to the office failed to fix it, so I and my colleague left, tails slightly between our legs and confident we knew on which junior developer's head our wrath was going to fall.
Back in the office we confronted the developer. "did you change anything?", "No".
"did you change anything?", "No".
"did you change anything?", "No".
"did you change anything?", "No".
"DID YOU CHANGE ANYTHING???", "well, yes, but that shouldn't have made any difference!"
We went back the next day with the working version.
I've always emphasized my desire for the correct answer the first time by an application of a suitably overpowered ShockyStick(TM). It tends to cut down on the repetition when the person being interrogated finds themselves writhing on the floor in an uncontrollable spasm of pain.
"Let me ask one last time. Did. You. Change. Anything?" *SparkZap SparkZap* (Screams for mercy) "Thank you for your prompt & truthful reply. Please remember this lesson for future reduction of unnecessary pain, yes?" (Mewling from the floor) "Very good. Byebye now."
*Happy sigh of torture memories from days of olde*
In my days as a Unisys DBA supporting a CODASYL database the following conversation happened at least once a month.
"The database broken!"
Not as far as I know, it isn't. Why do you think it is?
"Well I know I stored such-and-such a record on the database yesterday, and now it says the record is not on file!"
Did you delete the record?
"Of course not!"
Did someone else do that?"
"No-one would touch my test data!"
Lucky you. What did you change in the program?
"Nothing!"
So if I do an @PRT,S of all the library elements, the dates will all be at least a week old?
"Well, I DID change something in a subroutine, but it wouldn't effect the program!"
Let me guess. You added a few columns to your subroutine's DATA DIVISION.
"er ..."
You have displaced your calc keys by however many characters in your LINKAGE SECTION. You are chopping the front of the calc key off. That is why your record is not found, assuming your "no-one would touch" assertion is valid.
"So ..."
So you need to go fix your program so all the bits match the one bit you changed.
It got so bad at one point that when someone offered me a listing of their botch job and insisted the database was broken I offered a wager, that if I went through their code and found no cause for concern I would give them a crisp ten dollar bill, but for every protential problem I found they would give me one dollar. I pointed out that before they took the bet I could see I was five dollars up on the deal from what I could see on the first two pages.
That and the old "Your record counts say there are 66 thousand records on the database, but I can only find 60" thing. How many times did I have to tell them to re-establish currency when switching from "in set" to "in area" semantics? Well, I quit before an answer to that was available.
Been a nuts 'n' bolts engineer for decades, everyone I have ever worked with aside from the newly qualified, have lived by the code of ' if it ain't broke don't fix it'.
We always used to tell the newbs, 'You are the most qualified here because you have been taught all the latest stuff', hard to enunciate when your tongue is firmly in your cheek.
The newbs believed us and always thought they knew better until they realised they didn't.
... ask me how I manage to purchase heavy equipment for pennies on the dollar. A couple of quick bearing changes, some light welding, seals and the odd valve replaced, all fluids & filters changed, followed by properly adjusting all moving parts, giving it a bath and some appropriate touch-up paint and I can usually flip them for four or five times what I paid, including parts. Sometimes quite a bit more. It's more than enough to pay for my steam engine fetish.
We always used to tell the newbs, 'You are the most qualified here because you have been taught all the latest stuff', hard to enunciate when your tongue is firmly in your cheek.
Which is also why the newbs got to visit the CEO's home on Christmas Day for to setup the new computer for his children.
""if it works, 'enhance' it until it stops.""
Many years ago an old timer explained how a domestic radio was designed.
First of all everything was done according to best design practice for decoupling etc. Then components were selectively removed until it stopped working. Replacing that component established the economic production design.
There was a standard piece of advice if your HF transmitter PA had parasitic oscillations. "Add a parasitic stopper**. Unless you already have one - in which case remove it."
**this was a few turns of wire wound on a resistor body. It was in series with the power feed - close to the PA anode cap.
And good days for releasing life changing, Great Games altering news are over the Xmas period through to the New Year, for before anyone realises it, IT and AI will have realised it and moved on further into other stupendous fields in support of the latter which has morphed and now becomes recognised as the former.
Do you yourself discover that normally to be the case during such times of widespread merriment and jovial celebration, thus to find out the majority are at the mercy of a very select few one has no idea of?
Of course I know a few ones control many. Been doing it myself since 16, I believe.
The news changing game pause may be related to a certain inquiry into who is behind the phone not-so-legal inspection/infection - as he is attached to many others, and considers me a cockroach which never dies ( despite all the attempts to kill such a person ).
Alliances are formalized with rings, work and joy at the sea.
After the removal of the last tooth-microchip of MkUltrAI project, Elon, Bill and Zuck will suffer with lack of new ideas for projects. Take good care of them.
Selling to others the tooth-microchip of MkUltrAI is an international game-changer, isn't it?
With a functioning sample, even better, so tests can be done and all parties will be happy.
I am pretty sure I will be alive due to manly alliances/partnerships/proposals in place. I am a man of my word, unlike others.
I thought so speaking as someone who owned 2.5 of them
A500
A1200
A1200 with 68030 processor card - it was still the same host machine but with a massively faster processor doing the work.
I had to swap to Windows when the software support died after commodore went bust.
This is the rite of passage that nearly every engineer (and many others, including their PHBs) must go through _at least once_ as they learn how to function in the real world. Some people may need more than one lesson.
"Experience is a hard teacher because she gives the test first, the lesson afterward."
In my case, the update was performed elsewhere. On New Years Eve. Just before midnight.
An undocumented change to the data format in files coming over from an external company caused billing to fall over in a very large heap. Muggins gets the call and I spend my time coming up with a fix (because there was no-one available at the external company, and even if there were it would be unlikely they would be able to fix it quickly).
Fun times.....
When I was working as a contractor for a car manufacturer something similar happened.
A third party changed a file format with telling anybody ("I thought you would notice the format had changed"). Well, sorry mate, we don't eyeball your files to see if the format has changed but we did notice when the company started getting rapped over the knuckles by the local Customs authorities for thousands of undeclared cars assembled overseas (cheaper labour costs) because the electronic customs declarations for the cars failed, the shipping company started charging for the extra port fees and the time spent in port unable to unload the cars and the subsequent logistics snafus meant delivery dates for the finishing plants and dealerships slipped.
JIT is fine until an idiot throws a spanner in the works.
In my first IT-related job, a summer trainee helping to maintain a Honeywell mainframe, the late Friday evening was reserved for a reboot. This was done always just in case. I guess someone had figured out an intentional reboot once a week minimised the risk of an unintentional reboot during the working week. The mainframe had no battery-backed clock, so the official time for the next week was as accurate as the watch of the operator who entered the current time at boot.
Many years ago (early eighties) I worked on a monitoring and control system at a customer site. Nearby was a pub which served good food at lunchtimes; it also had a micro-brewery that provided the local ales. The customer representatives, my team and I used to go there often.
After some (trying) months, we had the system in and fully acceptance tested by the customer's team.
Of course, the senior management (from both my employers and the customer's organisation) swooped into the final hand-over [to take the glory for the help they'd singularly failed to give]. One of my manager's managers said with great generosity (and some condescension) that he'd take care of lunch (even though we knew what we and the customer particularly liked / avoided) and proceeded to make a spectacular cock-up of ordering food and drinks.
At which point my opposite number on the customer team piped up with "Bloody hell ####, when your guys said you couldn't organise a piss up in a brewery, I didn't believe them, but..."
The glare on the manager's face, and then the dawning realisation as he saw the fermenting vessels through the glass wall was something to behold (and never forgotten).
Name redacted to protect other innocents who were around at the time.
A UK MP** was offended when someone said she couldn't organise such a proverbial piss-up. She then organised a trip to a brewery and invited all her colleagues. Unfortunately she sent out the wrong date.
**Google finds many instances of politicians not being able to organise the proverbial.
Many years ago a prestigious bank bought a large stake in a proposed online transaction processing system. Customers were on remote Teletypes either by modem or wired telegraph connections.
There was to be a demonstration at a VIP jamboree at the bank's palatial headquarters. The night before - telco engineers were crawling over rooftops at the headquarters - trying to find the break in the temporary telegraph line for the terminal. Luckily they had solved the problem by the time the demonstration was to start.
The terminal operator was very experienced and she had been told to keep to the script without any deviation. This was because Plan B involved having all the query responses on a reel of papertape on a Teletype at the computer site. In case of a system failure - this could be patched in.
Against all previous odds the system didn't fail - until immediately after the successful demonstration. It took several days to get it running again
Just before a demonstration to various funding VIPs - the mainframe system wouldn't load. Fortunately it was a foggy day and the VIPs' aircraft was diverted to another airport - so they completed their trip by a slow coach. By which time we had finally discovered that the mainframe operator had done "a little tidying" on the filestore before closing down for the clean start reload. A very close call.
And people wonders why VMs just keep getting more popular. If you have to run a VM to get that program you have been using for years because it just works fine then you are gonna run a VM. That's why Windows 7 included a free Windows XP VM in the most expensive versions; you just had to download it from Microsoft website.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
