Intel seems really interested in pushing bad quality parts to their customers, are the Managers short on Intel?
During April and May, Intel started updating its processor documentation with a new errata note – and over the weekend we learned why: the Skylake and Kaby Lake silicon has a hyper-threading bug. The erratum is described in detail on a Debian mailing list, and affects Skylake and Kaby Lake Intel Core processors (in desktop, …
I remember (many, many moons ago, when I worked at a certain cellular base-station manuacturer that used a stylised bat-wing for a logo) having an arguement with the technical architect about the use of non-Intel processors (AMD had just bought out the K6 and I was quite keen on them).
His answer was to specify that non-Intel processors should never be used "because you knew what you got with Intel and their processors were bug-free".
The next day Intel revealed the news about the FooF microcode bug.
Oh, how we laughed.
I can't access the computer unless/until the ScreenReaderEnvironment (SRE) starts talking, & there's no pre-OS-SRE, meaning I can't boot the computer to a SRE in order to get into the BIOS to update this kind of crap even if they DID release a fix that was SRE accessible.
I'll have to pay a Sighted Techie to do this for me, and that means heading to someplace like BestBuy.
I *really* don't want anyone at BestBuy touching my computer, I don't trust them as far as I could comfortably shoot one out my arse. =-(
*HeadDesks in frustration*
You're going to have a long wait if you want OS X or Linux executables to patch your UEFI.
For Apple Mac users that's just another patch to run (see this as an older example. Firmware BIOS or UEFI upgrades are generally not hard to do, but they're rare. Other than with Apple where it simply shows up as an update that needs a reboot to complete, few users realise they need it because it's not really part of the Microsoft or Linux update cycle - it's hardware.
Unless users are in touch with whoever manufactured their hardware, how will they find out? I guess it's easier with Microsoft sourced hardware where the associated subscription will at least provide a point of contact.
For Linux, you just need the package intel-microcode.
You get it with 'sudo apt-get install intel-microcode', or graphically with the dialog box for additional (aka proprietary) drivers. You would use the same dialog box, for example to get the nVidia proprietary drivers instead of Nouveau which is the Open Source implementation for nVidia GPUs.
Here is what the documentation of the intel-microcode package says:
This package contains updated system processor microcode for Intel i686 and Intel X86-64 processors. Intel releases microcode updates to correct processor behavior as documented in the
respective processor specification updates.
So you just need to wait that you distro updates this proprietary stuff.
This post has been deleted by its author
intel-microcode might help in this limited circumstance (assuming it handles suspend properly) but what about other issues that require UEFI patching, like the recent ME exploit? That can't be fixed through the Linux kernel since it operated below the level of the OS or even a hypervisor. Any laptop that doesn't let you patch UEFI directly that is running Linux needs a Windows partition - that's the only reason I have one on mine.
If you could fix the issue if Screen Reader were working then I am sure you could find someone willing to act as one while you accessed your BIOS.
While sympathising with your predicament it surely isn't the first time you have ended up needing reading assistance which technology doesn't cater for.
I stand corrected on OS X UEFI updates. I hadn't ever seen them on Intel's site, but it makes sense Apple would distribute them directly.
You still can't get Linux executables to patch UEFI, if your UEFI doesn't patch itself directly like my laptop's doesn't, you have to keep a Windows partition around just for that purpose.
did... you seriously just tell a blind person that you're "sure" they can "find someone" to just be a screen reader?
if you *really* do sympathise with the difficulties that come with being disabled, then please, heed my advice: comments like the one you just left aren't helpful, and in fact, they are patently unhelpful.
this person quite obviously knows their living situation and everything associated with it. if there was someone who lived locally enough that would help, that the person trusted, that they would do that? instead, they said their *only* option would be taking it to a store, which they didn't trust with their computer.
which goes to say that if there were a generic Someone they could ask "to be a screen reader", they probably wouldn't trust that they were only doing what was asked of them too.
let alone, like... i'm guessing since you post on this site you've had the "joy" of doing tech support for uninitiated family members over the phone? ones who describe what's on screen in such a bizarre way that you have *no* idea what's going on? that's almost certainly what it would be like for this person with a generic Someone, even if such a person were available.
now, i'm going to assume good faith here, and bring up that: a lot of disabled people (or people with disabilities, depending on what the OP to this chain prefers [people do often prefer one or the other, i usually go with "disabled" for myself rather than "person with disabilities"]) do live independently enough that family and friends don't live anywhere near close enough to just come round and offer support like this on a whim.
i bring this up because it's still very prevalent in our popular culture the idea that disabled people always have family, or a romantic partner, or some kind of hired carer, available at all times. which is simply not the case for a lot of people (even though there are people who need such 24/7 help) and as such, is generally pretty patronising to say something like what you said, that you're "sure [they] could find someone". why? *what* makes you sure? please take some time to think about that.
disclaimer: i am sighted, but disabled in other ways. i do not presume to have all the answers of how to speak to blind/Blind people specifically. rather, i am just painfully aware of how people with all kinds of disabilities get given "advice" like "get a friend or neighbour to help you with it", as if it should be normal for someone to just need the charity of an abled person to do something everyone else can do independently.
I think the most important point you missed is that disabled people* generally don't want to have to depend on other people. Independence is something very powerful and precious to them.
*Not sure what the correct phrase or terminology is here, sorry. No offense intended.
Facetime with a sighted friend who will read the screen to you.
There's an iPhone app that senses light, so you can even tell when the screen lights up.
(Equivalent tools are no doubt available on Android and Windows Phone. It's just that all my blind friends use iPhones.)
Well, have you seen those symptoms yet? If not...
Also, look at how long it took them to find this, since it affects all Skylakes it is at least two years since customer testing of Skylakes started, even longer since Intel did internal tests. It can't be all that common or easy to trigger, or it would have been identified a LONG time ago.
Well, on the other hand, how many people actually bother (or are able) to root-cause random crashes to the Point where the microcode can be properly blamed?
Even if it's a perfectly repeatable crash most people would blame it on application/OS bugs and work around it.
I'd think that the only people who actually HAVE to hunt down bugs like this would be compiler authors and related fields.
Not necessarily. Guys who run performance benchmarks at Tom's Hardware Guide once noticed that newest Intel Pentium III 1333MHz (or was it 1133MHz?) consistently crashed at Linux kernel compilation test.
Found it on Wikipedia: "A 1.13 GHz version was released in mid-2000 but famously recalled after a collaboration between HardOCP and Tom's Hardware discovered various instabilities with the operation of the new CPU speed grade."
"When I was testing the Pentium III 1.13 GHz I was also using a GNU/Linux installation to run kernel compilation benchmarks. I had introduced this benchmark in our processor-benchmarking suite only recently and was therefore not too experienced with this operating system. What I knew for sure however was that my Pentium III 1.13 GHz hadn't been able to finish the compilation even once."
I agree fully. Looking at the timeline in the Debian message, this has been ongoing for some time. So incidence and significance seem favourable.
And as for some comments here:
The latest Dell boxen we have here allow BIOS flash/updates OS independently, because there is the f12 option way before any OS has booted. Just get the update package at Dell and use the update option under F12 during boot. Whether Dell already adapted/ upgraded their BIOS, that is another question of course (quick check for our boxen resulted in a firm no).
As for Linux - not uncommon, the Linux updates are also coming through with a significant delay. For example, where Debian (from stretch upwards) already have the new intel-microcode, ubuntu still haven't pushed this newer version (for their LTS) yet.
Then again, as WUSP show, careful review and deployment of updates is always a good thing...
This post has been deleted by its author
I have had some unexplained major performance issues with a heavily multi-threaded application on a Kaby Lake i7 that do not occur on my Haswell i7 development machine. At times the 3.6GHz Kaby Lake significantly under-performs the 2.2GHz Haswell. I kind of doubt this microcode issue is the cause, it's much more likely to be a bug in my code, but I have been racking my brains and not found the cause yet, so I'm going to disable HT and see what happens.
> "or it would have been identified a LONG time ago."
It may well have been found a long time ago. Trouble is, intel have a habit of sitting on information like this so as not to harm their channel partners (who would have to dump stock), and themselves who may have to issue a huge buyback.
Better to wait and bury the head in the sand while the engineers can figure out if it can be fixed in the field.
I have a laptop given to me by IT sitting in the corner. As anything handed out by corporate IT it is an Intel. To be more exact Hell with Skylake. I do not recall a "hyperthreading off" toggle in the BIOS though...
I had to put it on extended leave because it was showing some seriously weird behavior with Java coredumping. I suspect it is the same bug. Based on its behavior I would say - every couple of hours under heavy load.
>Anyone know how likely this is to occur?
Agree, if I understand the problem correctly, it should be causing user visible random system glitches and instability. Thus given the number of PC's running Windows it is correct to ask why we aren't seeing a lot of feedback about random Windows faults etc.
Additionally, whilst I don't know the extent to which the OCaml compiler is used, we should also be questioning whether there something the OCaml compiler is doing (with respect to hyper-threading) that Windows and other compilers aren't doing.
Current Linux distros (Ubuntu from at least 15.04 on) have a "3rd party driver" feature to update the CPU microcode. Both, for AMD and Intel.
Does this solve the problem? If so, enabling that driver would be a simple workaround.
I wonder, if a similar feature is available for Windows, too.
Edit: See also:
Hmm, a short search of the microsoft website shows that there is at least a mechanism for microsoft to update the microcode. They seem to deliver it via windows update.
Perhaps Intel can persuade them to deliver this specific update quickly and with a comprehensive description?
"Current Linux distros (Ubuntu from at least 15.04 on) have a "3rd party driver" feature to update the CPU microcode. Both, for AMD and Intel."
Such mechanisms have existed since the days of oops-I-can't-divide. So why are Debian saying it can't be fixed except by motherboard firmware?
Does current firmware shut the door on such mechanisms? That might be done for security reasons - block malware that attempts to rewrite microcode - but if so there needs to be a better way to fix it than depending on motherboard manufacturers getting round to distributing upgrades, always assuming they can be bothered.
This post has been deleted by its author
For Skylake, the latest version of the Linux microcode update will fix this.
But for Kaby Lake, Intel have not released the fixed microcode publicly, it's only available to motherboard manufacturers. So you will have to ask your PC or motherboard vendor for an updated BIOS/UEFI that includes the latest microcode.
That's Intel's fault for having a fix and not releasing it to the public, there's not much that Debian can do about it.
Don't get all giddy. Apple will deliver these new systems with the microcode already patched, you can be sure of that.
They don't have to. As soon as the user has done first setup it will run the update and pull in the relevant patches since production, update, possibly reboot and that will be that.
Sure, what is produced now (well, in a few days from now) may be supplied pre-patched already, but there's no point in opening up existing stock, that would be silly :). There is also the problem that something new may have shown up by then :)
The Screen is a whole lot better as is the GPU than the 2009 version but you don't seem to care.
And the problem is probably temporary so won't affect many people (who buys iMac's these days eh?).
but you seem to really not like Apple so carry on hating them even though this problem does not lie with apple but Intel.
"So the new £5000 iMac Xeon 2017, is well, the same speed as a 2009 i7 iMac 27''?"
It will have eight cores instead of four, at higher clock speed, so it will be an awful lot faster.
It is also supposed to ship in December 2017, and we might assume that Intel will get its act together and ship a fixed product by then.
It will be great.
The screen is the same as my iMac 5k, so very nice.
My iMac does have some cooling issues.. if I compile for 5 minutes, it will get quite warm and the fam will spin madly, I wonder how much noise/heat issues will de Xeons have.. will they throttle while rendering?
As for rendering, etc.. well, I would rather have a threadripper, much faster and cheaper. But sure not as pretty, and if you like Mac OS...
This can only happen when both logical processors on the same physical processor are active.
Combustion can only happen if air contains oxygen
In about 1981 a computer manufacturer - not Intel - trained me and a colleague (we were in an outside company) to create new microcode for their processor.
Microcode is hard. Components of the CPU are running in parallel, some taking several clock cycles, and the microcode designer must take account of these things.
I am sure modern Intel CPUs are much more complicated than anything from 1981. I hope they have better development tools, including simulators, than we did. But now I wonder if there are bugs or design weaknesses in those tools.
Or, I suppose, it could just be undue timescale pressures on the microcoders.
In 1966 the RCA Spectra 70/45 mainframe had microcode that could be software selected for either IBM 360 compatible mode - or second generation RCA 501 (English Electric KDF8) mode.
The ICL 2903 mini-computer range was programmed at microcode level.
The IBM 370 compatible mainframes like the Fujitsu/ICL Atlas range in the 1980s had microcode that could be loaded from floppy disc. This offered various processing speed upgrades as and when the customer required them - without major disruption.
"Which reminds me that I should probably re-read Tracy Kidder's excellent book on Eclipse MV/8000."
Somewhere in that book is a quote along the lines that at least one person afterwards "went back to the soil" - where things moved more slowly than nanoseconds.
Having taken a break from IT on a farm for similar 1970s reasons - I was then glad to get back into IT "where things move a lot faster than the seasons".
The book's subject of the unofficial "backup" development of something also happened in ICL - IIRC with the comms front-end MCU1. Can't remember why the approved new development failed to meet its deadlines - but a skunkwork based on existing hardware was a success.
The problem occurs only with active hyperthreading, i.e. when the CPU is executing two different instructions in parallel that are *supposed* to use non-interfering components of the processor. Hard to find that with simulators.
The code pattern to trigger this bug is also discouraged by Intel Optimization guides as being slow, so probably only coded very rarely by a compiler. Probably not a high priority on simulation experiments until now.
Or 'We think there's a problem around this, let's discourage people from doing it'.
One of the issues is apparently that Intel test their processors less than they used to. Taking time to make sure they were behaving correctly was costing money and making them look slow...
..for the random blue-screens and freezes that nearly every Windows box seems to experience in our organization once in a while. Issues that have happened to our newer Dells with i5s and i7s and that we've never been fully able to eradicate, despite patching/updating everything possible, swapping memory, hardware, etc.
No excuses or benefit of the doubt though for the utterly awful Windows update process that needs to be taken out back of the barn and shot.
Sorry to ask, but have you followed the proper well-established procedures for troubleshooting bugchecks? Probably not, since you can't even tell if this issue can be related or not.
I mean - you are aware that those letters and numbers on the blue screen are there for a reason, aren't you? (Though for serious debugging you will want a kernel dump as well...)
I happen to have a rather nice* Dell XPS 8920 at work. Last BIOS is from March. There's no option to disable HyperThreading.
So far the only things crashing on me are PulseAudio and bluetoothd, and I have no idea if they crash because of this bug or just because there are bugs that need to be ironed out in drivers or the software.
*It's nice after putting up a good fight when I installed Ubuntu on it. I had to mix and match many Internet forum posts in order to win the battle.
I'm just schocked Nostradamus has not even vaguely reported this. We all know the Pentium FDIV bug was a big hit in 1520, so why is there not even a teensie-weensie Quatrain about this one? Not even Hoagland has anything on it (as of this writing). Are we looking at a new conspiracy, stemming from like 500 hundred years ago? (That would make it a 2000-Quarterly financial report Head Fake; must be a record :)
So now AMD to sell Ryzen has to attack Intel finding bugs that highly like don't exist...
Everything about this new microcode bug affecting Intel CPUs looks beyond fishy. And AMD needs to sell the awful Ryzen products at any cost to not go bankrupt.
So they come up with these lies to attack Intel trying to tell the world "you see Intel CPUs are not more reliable than ours" ... What a smart masterplan ..uhh! Fact is that AMD lies against Intel and Nvidia always end up being a huge failure. They think everyone buying CPUs and GPUs must be dumb like their so smart marketeers ...
Heck ! AMD sold the Polaris GPUs out of PCI-E specs that fried the motherboard against any safety standard and then they released a software "fix" with their drivers to slow down the Polaris GPUs to avoid the melt down. THAT is AMD , the ones selling unreliable flawed products.
Not a word about AMD in the article. Nothing mentioned about GPUs.
Yet you bring up both topics. And then you add a conspiracy theory about AMD being behind this.
While I understand that this may be simple trolling or fanboy knee-jerk reaction, I still feel compelled to ask: Why did you actually type that flame out?
I've experienced a lot of crashing of a Skylake i7 that is resistant to BIOS, driver and OS updates. My research lead me to several very long intel discussion forums of people experiencing similar crashing and not much help from Intel. So it's nice to see this is finally being acknowledged at least. Unfortunately the Asus m/board I have seems to not always puch a BIOS update when intel publish microcode updates/fixes, so it could still easily be well over a year before I get a fix (if at all). At least I have a potential working solution now (disable hyperthreading), previous suggestions (on the forums) have not worked, e.g. disabling the various CPU power management strategies has been one most discussed.
You can load microcode after boot as well, from your favorite OS. Linux has a tool for it that runs on boot - t Think it's included with Debian but the actual microcode isn't since it's a non-free binary blob.
Windows has it as well and with some luck microcode updates arrive via Windows Update.
Doing it at the BIOS/UEFI level is mostly needed when a bug/missing feature prevents the system from booting at all.
I just did a Mini-STX build with a Pentium G4560 and notice that the system crashes regularly due to the integrated GPU if I don't enable the proprietary microcode. Guess I'm just damned if I do, damned if I don't at this point...
You should ALWAYS update the Microcode, running without any Microcode patches is really really bad.
All processors are designed to have fixes applied as bugs are found, instead of having to "mask out" a new processor DIE each time, or slowing the process of releasing a new processor to 1 per decade.
This is not a design flaw, it reduces costs if they can fix design flaws on the finished product after it is manufactured or even after it ships.
Dozens of major and minor bugs have been fixed in the Microcode update prior to the first processor shipping, so running without these fixes is really really really bad.
When a bug is found during the extensive testing phase, a Microcode fix is designed to resolve that problem. And while we could complain about this not being found earlier, it is usually the support community that finds these obscure hard-to-find bugs that feedback into the CPU company for fixes.
Consider it the same as never updating a software package, even though known bugs are in it, without uninstalling the original software package and downloading and installing a whole new version.
You would be asking why the software company didn't just release a patch to fix it.
Why wait for a bios update or perhaps a microcode update via windows update?
With this VMWare thingy you can grab the latest microcode files from Intel (and AMD, otherwise the program may not work - not tested). This is installed as a driver, and in the event log you can see if a microcode update has been applied.
Use these microcode files:
Also works great for stuff with no bios update available.
AMD Processors have Microcode Updates released regularly as well.
In fact, when I was designing AMD systems based on bleeding edge chips, like Ryzen, there might be two or three DOZEN Microcode updates after the CPU was released, and perhaps dozens to hundreds of patches already in the Microcode update.
Every time we released a BIOS update for our AMD systems, we would get the latest Microcode update from AMD.
All CPU manufacturers have Microcode updates and issues like Intel just had.
Don't use this as a justification of the difference between Intel and AMD.
I use the fact that AMD "Invents" technology, Intel "evolves" technology.
The first brings us new cool stuff at much lower costs, which the second is safe and profitable.
If Intel was smart, they would use AMD as an R&D house and use Intel's manufacturing capability to build them.
I am working with Intel Support related to a bug I was able to reproduce repeatedly.
They suggested changing Graphics Settings in the game that exhibits the problem.
While this did help, eventually lock-ups and even a corrupted settings file began happening.
Turning off "Turbo-Boost" appears to solve the problem
Reading this article, I have now turned Turbo-Boost back on and turning off Hyper-Threading instead to see if this solves the problem as well.
Since turning off Turbo-Boost seems to fix the problem, I am wondering if that is a better solution than turning off Hyper-Threading.
According to the conditions that "should exist" for boosting to happen, it seems pretty rare that boosting "should" actually happen and losing half my processor threads seems like a bigger performance hit.
Thoughts? (please not rants)
1) Should turning off Turbo Boost also fix this problem? (doesn't seem like it would, but appears to in my case)
2) Would you rather turn off Turbo Boost or Hyper-Threading to solve this problem?
BIOS UPDATE METHODS
If you download the Intel BIOS File onto a USB Flash Key, then:
1) You can update the BIOS in the built-in BIOS Setup utility by surfing to the Device and Directory the BIOS image file is in.
2) Can do a "BIOS Recovery" by holding down the power buttom from the Off state for around 8 seconds (no longer, a LED color change usually signals the recovery) but the ONLY file that can be on the USB Flash key can be the BIOS Image file
3) Use the motherboard vendors BIOS Update utility
So for Linux users, there should be at least 2 update methods available without getting someone else to help you. But it is possible that the BIOS vendor or modifications specified by the motherboard vendor do not support these options.
Biting the hand that feeds IT © 1998–2022