BOFH: Have you tried forcing an unexpected reboot? BOFH logo telephone with devil's horns There are few things more annoying to an IT Professional than applying a firmware update that crawls to a stop at 83 percent. Luckily, today, I have one of those more annoying things right here with me. The Boss is peering over my shoulder. If there's one thing that can make a bad …
Nah, this is actually an example of an Einsteinian relativistic task completion distortion field. This theory states that time moves at different rates for those carrying out a task and those observing the task. The closer to the end of the task, the slower time passes for the observer. This explains not only why progress bars appear to drastically slow down above 90% and get stuck at 99% for an hour, but why the arrival time display at a bus stop showing "1 min" means that the bus will arrive in 5 minutes for the people waiting there, and programmers promising finished code "by Friday" appear to be still working on it 3 weeks later.
I think that only explains the start of the situation. Once you determine the update procedure has, er, bottomed out, it all goes quantum.
The repartee with the engineer over the model number and whether or not to open the case and obtaining photographic evidence has the ring of experience from dealing with UTEs – the "unqualified technical engineers" who plague our existence!
Round North London possibly other places- 4 buses from different routes, but covering the same stretch for the next few miles, should in theory arrive spaced out about one every 7 minutes (averaged), but will somehow all manage to arrive at the same time, then leave a half hour gap until the next group.
There is a rational explanation for buses all turning up together.
If Bus A is running just a little bit late, then at each stop there will be a random increase in the number of people getting on (and later off), and hence each stop takes that little bit longer, so the bus gets later and the passenger at each stop that little bit bigger.
Bus B following behind now has fewer passengers, and so keeps to time or in extreme cases starts getting early, and so gradually catches up with Bus A
This continues until Bus B catches up with Bus A, and then they start leapfrogging each other, which delays both of them.
Until Bus C catches up with them
Repeat as required !
Which would make sense, except where three or four buses go up to the terminus at 10 minute intervals, but emerge 40 minutes after the first one together Something you'd only be aware of if you happened to be catching a bus a few stops down from the terminus. You wait for the bus, one goes up, so you assume that one will soon come down. But then another goes up a few minutes later, but still nothing comes back down. Then another, and another. And you wait. And eventually they all emerge like Daleks, one behind the other.And it keeps happening- sometimes I'd only see two or three go up, but still four would emerge. Presumably one or two had already gone up and were waiting for the rest of the club by the time I arrived at the stop.
You reminded me. TfL always claimed that London buses arrived bunched because of the effects of traffic. Just unlucky.
Except, I used to get a bus a few stops from the terminus. Maybe 5 minutes away.
I'd see a bus go up, towards the terminus.But none coming down. And I'd wait, and ten minutes later I'd see another on the same route go up, then usually a third and often a 4th. And I'd wait, about 30 or 40 minutes later they'd all appear like Daleks in a row.(It's why I stopped being a Good Citizen and went back to driving- reduced an hour+ journey time to about 10 minutes, which saved at least one whole teaching slot each day and left me much less stressed and considerably more reliable).
It's the classic case of the last 10% taking (at least) 50% of the total time. Personally, I'd say the last 10% takes 80%.
For example, I needed a simple Raspberry Pi-based app for timed opening and closing of relays. Half an hour, including testing and bugfixes for a basic version, another couple of hours for one which did different things according to the day of the week or the date (no relays required on Christmas day, for example). Not counting the odd bug which only appeared on certain dates or after a certain amount of uptime.
Making the on-screen display "pretty"?
Two days. And another day of tweaks a couple of weeks later.
Making it easy for someone other than myself to update the timers?
Erm... still working on that one, three years later.
Same sort of thing happens at home, so I've learned to overestimate. Yes, I can definitely mount slatted shelving in the airing cupboard and fit a folding door. SWMBO has a stack of fresh bedding and towels piled up on the landing and expects to get them neatly put away before tea time. Even with a willing (and able) minion to help, the job actually takes a very long day, by the time I've been to the DIY store (oh, and picked up groceries as well), measured, cut, mounted, eaten lunch, had several cups of tea etc. etc. Would have taken even longer, but this is the third folding door I've done now, so I don't have to waste half an hour with a magnifying glass trying to decipher the badly photocopied instructions. Without the minion the job would definitely have run over into the next day.
When it finally became time to stop using the downstairs loo as a spare storage cupboard and fit it out as a loo (we already had the loo, the sink, the taps etc.), I had to break it gently to SWMBO that her estimate of one long day, or a weekend at most, was at least 50% out. It took three very long days in the end, even with minion help.
M.
There was an old saying, that a project would progress to 90% completion, in 90% of the budgeted time, and the remaining 10% would take the other 90%.
It probably was actually accurate back then - in the distant and murky past, when I was a lad etc etc..
These days everything is bigger and better, so stick a couple more orders of magnitude on the time factors..
I read that as
The Boss is peeing over my shoulder
which gave an interesting slant to the rest of the tale.
My bugbear is progress indicators that show 100%. If it's 100% then the update is finished and the device should be either rebooting or now available for use not still showing the progress indicator
Then there are the ones that reach 100% and then go back to 20%. With a bit of luck they move forward again.
Yes, I'm thinking of Windows updates. That's why I prefer to do Linux updates from the command line - you get an actual commentary of what it's downloading on a file by file basis, what the current download ed amount is and they speed before moving on to what it's unpacking or setting up. You might not know what all those library packages are fore but at least you know progress is being made.
A place I worked for in the late 90's had a CRM application we regularly had to install on agents' PCs. The installation progress bar didn't have a number on it, but it continued a couple of notches past the end of its predrawn box, so when someone had problems we always used to ask if they'd remembered to install the full 107% of the application.
Getting ready to leave for the weekend, train leaves in...oh, around 30 minutes.
Click on START, shutdown, UPDATING, DON'T TURN OFF YOUR COMPUTER!
[oh, sh*t]
Whoever the numpty is at Microsoft, who thought updating should be done when the computer is switched off, should be made to miss a few appointments. Update when I turn it on, while I'm on the clock.
"Sorry Boss, Windows is updating..." is much better than "Hurry up, you damn machine, I have a {train|girl|guy|beer} waiting for me!"
Got my coat, but can't go anywhere until the damn machine finishes updating...
My work week is essentially Friday to Tuesday at the moment. Nearly every Friday I come in, switch on the computer, boil the kettle while it boots, load up Outlook and Edge, start reading and replying to emails from the people who can't understand why I'm not reading emails on Wednesday or Thursday and then, usually as I'm right in the middle of a reply, maybe with a spreadsheet open for reference and several tabs open in Edge, a little box pops up - not a "Windows" box, but from some third-party update manager our IT department uses - telling me that it needs to close Office apps in order to update and will do so in five minutes unless I click "postpone".
So I have two options: race to finish the email before the timer's done or click postpone.
If I click postpone there is no option to "try again in 1 hour" or "click on this button when you are ready", you just have to wait for the box to pop up again, a seemingly random amount of time later, and again, nearly always while I'm in the middle of something important and not during the two hours I spent away from the desk on the "shop floor" as it were.
And then if the update requires a reboot, you might as well walk away from the computer for half an hour or more.
Why this blasted bit of software can't pop up within two minutes of starting the machine, while the tea is hot and undrunk and I've not started any serious work I don't know, and as for having to shut apps down in order to update them, or reboot Windows and spend hours looking at "updating" before shutdown, "updating" before bootup, then a reboot and another "updating" before getting control back! Coming from a personal use of Linux I have really become appreciative of that system's ability to update even running apps in the background, with the final stage of restarting an individual app or rebooting the OS for a kernel update left up to me. I can work all morning while the system updates under me, then reboot the machine at lunchtime perhaps.
M.
Trying to update the firmware on an old Marvell 9123 sata controller.
Not quite stuck at 83%, but one version says it applies, but no card or devices - OS boots with none
Another says applied, shows the devices but kills the OS
I know it is an old card, but it is needed for an old external box. Newer esata cards using ASM just lockup no matter what.
It's the ones with a "Time remaining" indicator that annoy me the most as they bounce from "5 minutes" to "20 seconds" then "20 minutes'
In the early days of Bluetooth the adapters came with their own bespoke BT stack as well as the driver as M$ hadn't got their own stack. That meant that, if you had to change BT adapters to test relative suitability for our products, there'd be an awful lot of uninstall/reinstall going on.
When uninstalling the WIDCOMM stack, the progress would get to "10 seconds remaining" and, 20 MINUTES LATER, that would change to "2 seconds remaining" for another 40 seconds..
The thing begins the update, then the screen goes blank. Without warning. With a progress bar rolling on the screen normally when it happens.
Then you hold your breath, and won't let go until the screen shows up again.
Then it skips straight to 100%, whatever percentage was there, it updates to 100% LITERALLY off-screen, er the screen was turned off.
All the spinning wheel shows you is the subtask that displays the spinning wheel has CPU cycles.
If the system locks up ( eg CPU loop else where) the spinning wheel does not get dispatched - and so stops.
So all the spinning wheel tells you is that there is not a CPU problem.
@ColinPa
"So all the spinning wheel tells you is that there is not a CPU problem."
I hate spiny wheel indicators. They dont tell me jack about what is going on and less useful than some sort of progress bar or even the percentage progress indicator. Certainly more comforting to see text scrolling up the screen
all the spinning wheel tells you is that there is not a CPU problem
In an ideal world, perhaps. But an update process that is I/O bound, as most are, is likely to run the spinner in its own thread, so it continues to update regardless. It "tells you is that there is not a CPU problem" in the sense that the CPU is still there, but that's a very small reassurance.
It's fairly common to be faced with a situation where estimating the size of a data operation (eg "SELECT COUNT(*)" with a complicated predicate) takes nearly as long as actually doing it. In those situations I have been known to display Zeno's progress indicator, where the bar regularly increments by a proportion of the remaining space, and so never reaches 100%. By contrast, Microsoft appear to have used random numbers for time remaining when transferring files: 30 sec... 2 days... 1 hour... 10 sec... 20 min.... This gave the impression that File Manager suffered from something like bipolar disorder.
I agree. I've never assume ( or believed) that any loading/installing/whatever animation was more than that. That if it was spinning or whatever this just means that the little minprogramme that makes it run is still running. Not that it's tied to the actual occurrence.
Every spinner/progress indicator I've ever programmed was in the thread doing the actual work. It hadn't even occurred to me to put it in a separate thread, since (a) as you note, it would cease to be a valid indicator of things locking up and (b) you spin up a new thread where there's a for-real performance issue and need to spread the effort out, and a progress indicator (properly written, anyway) is not a performance hog.
(I don't deny that you've seen situations where somebody put the progress indicator in a separate thread. I'm just saying that they did something pointless and stupid.)
(I don't deny that you've seen situations where somebody put the progress indicator in a separate thread. I'm just saying that they did something pointless and stupid.)
I'd suggest that in high-level language (something like C#), which is inherently multi-threaded, you'd make a progress-bar event-driven, which means that by default, it wouldn't be on the same thread as the one doing the stuff that needs to display its progress, which is quite likely to be CPU-bound, or disk-bound, or network-bound at some point, thus becoming unresponsive if it was running on the main UI thread. You'd not be explicitly putting that progress bar into its own thread, but the language's own thread management would be doing it for you.
For something low-level, like a BIOS update, the question is whether you'd know enough about the system it's running on, or its state, to make the assumption that more than one thread was even available. it depends entirely on what language you'd written that flashing process in. Arguably, you'd want to make sure the thread doing the actual work isn't interrupted in any way, so it might make sense to farm the progress indicator off to another thread, if you could assure yourself of doing so without it causing timing issues.
That works well, except for that when you do that the moment you turn your back it pops up a little message asking you if you are sure you want to do the Update. There is a place in HELL in the super-laxative department for those who thought those were a good idea. A progress bar is fine, but it needs to move at least a little every minute or so. Percentages are like weather forecasts, it's somebody's educated guess but really shouldn't be relied on.
This reminds me of a job I did a few years ago, firmware upgrade 2 switches. Did the checks loaded the new firmware and told the device to reboot.
The switch (2960x) was still unresponsive 20 minutes later so wandered down to the comms room and plugged into the console port
rrrrrrrwrrrrrrrwrrrrrrrwrrrrrrrw
It had a microcode update over an hour after the reboot it finally came back to life.
For some reason I didn’t do the second switch that night.
Still not as bad as rebooting a switch after deleting the old image and BEFORE copying the new image over, that one I had to upload the boot image via the serial port.
... but I've actually done this several times. I've done enough upgrades and images to know just how long ANY PC should take. It's always different, BUT, there is an outer bound, and when time goes past that outer bound, I do a hard restart. If it fails, I just start over.
Often, it's just M$ with its head up its ass and just needs a good whacking.
Yeah, and do not get those of us with scars started about lack-of-progress bars.
Well it should be a fundamental requirement for visiting engineers to check the firmware versions of all system components BEFORE replacing, say, one of the RAID controller modules that had failed with a replacement controller module that has a different firmware revision. You never know, the controllers might start systematically copying blank bits over the top of very important data, requiring someone to yank out the power cord before it erases everything.
You never know when you might need that backup tape.
I cannot BEGIN to count the times and number of places I've worked where checking for updates FIRST, is the last bloody damn thing the techs do, not to mention JUST working the solitary support issue and walking away when done instead of ALSO checking for updates.
Christ, I've even been berated for checking for updates.
My level of disappointment and disgust has no words.
I loathe adding/replacing drives in RAID arrays.
Never sure if it's actually rebuilding the array or deleting everything in sight - until it reboots.
Seen that happen. Never figured out if it was me or the firmware that screwed up.
Fortunately had the nightly backup tapes in hand.
the people who suggest not having the lift there at all, then the photosensor 1/2 way down the lift shaft thats set to detect the infra red signal of a passing body can then activate the panel that swings out to divert said body from the lift shaft down into the wood chipper.
And the first person I'd try it on would be the tech who came in a few weeks ago to change a HDD, and didnt bother 1. seeing if the manufacturer had loaded the correct version of the software and 2. applied the correct service pack to it (I think its at about 15-16 now)......
Still, distracts me from my current set of problems.... engraving sequential serial numbers on the bits with leading zero suppression and the correct justification so the last number ends up in the same spot each time and the added bonus of the robot not remembering the current part number on power down unless its written to a data file, and then adding a macro routine so that 1 in 10 parts can be checked by the robot with in-program adjustments and automatic replacement of worn out tooling, to which the QA department said when they found out..."we want a measuring log added too" despite the fact that they'll not even look at it when the robot is running...
And all because the min wage operators we get prove time and time again that they are only worth min wage. can I retire now... please... pretty please....
We were all PFYs once.... then our elders left/retired/died
Still..... theres always the news headlines to consider
"Plucky woman makes it in a manufacturing world"
although its 50% likely to be :
"Woman mass murderer imprisoned for life in Broadmoor after chainsaw rampage in factory"
"We have here a Schrödinger's firmware situation,"
"The box ... it is both dead and alive. However, once I power cycle it, it will be one or the other – and most likely dead."
In the Many Worlds Interpretation there must be one world where every firmware update (ultimately) succeeded which is possibly the best evidence against that interpretation.
Although I have noticed once I have finally got one difficult firmware (bios) update to finally work identical hardware just falls into line. Sheldrake's morphogenetic fields?
"Hung firmware updates" are one of those things where you just have to leave the thing to it late on a friday and see if it's finished on monday morning (only downside is that 50% of the time you'll find it's asking for some stupid confirmation 25% through the process and you'll have to wait another day for it to finish). Watching a firmware update complete is a near guarantee that it won't actually complete and end up in a state of superposition. For some odd reason firmware updates are opposite to particle physics such that observing the outcome actually CAUSES the waveform to coalesce and stabilize. Firmware updates must therefore be done completely unobserved and you'll find it collapsed to either the failed or successful states once it's passed the firmware event horizon (warning, crossing of the event horizon MAY be accompanied by loud bang and smoke)
I once installed a macOS update on a friend's laptop. The damn thing went all the way through the update process, got to the whole "finishing up" part of the performance that comes right before it reboots into the updated OS... and then just didn't. Didn't finish up, didn't reboot, didn't do anything. Just sat there with its skinny little progress bar at 100%, chewing on something.
For obvious reasons I didn't want to risk interrupting the process of completing the OS install. So I sat there watching it do whatever it was farting around doing for a full 90 minutes before I finally threw in the towel and forcibly power-cycled it. At which point, it came right up in mere seconds.
Great and all, but man was I pissed off that I spent AN HOUR AND A HALF watching the thing do... absolutely nothing, as it turns out.
I have had my share of failed updates while in IT. I always turn purple holding my breath while doing these. My last failure was my own laptop. I had a replacement sent. They used to have emergency BIOS recovery option on older PCs. Why there appears to no longer be this option is beyond my own capability to render. My new laptop had two SSDs and I keep my data on second drive. I also do periodic Image/Data backups.
I wish all the best when these situations arise.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
