Hidden Linux kernel security fixes spotted before release – by using developer chatter as a side channel Boffins affiliated with BMW, Siemens, and two German universities say they can pinpoint obfuscated Linux kernel security fixes, developed in secret, before they are officially released. This is insight miscreants could use to develop and deploy exploit code before patches are widely available. What's more, the team found that …
It's difficult to work out what this article is trying to tell us. If I understand it correctly there is a theoretical possibility that vulnerabilities can be determined by public traffic analysis. Is there any evidence it has happened? At least everyone knows the potential weaknesses.
What does security by obscurity offer?
Au contraire... it seems that burglar alarms might actually make things worse.
Having read the paper you linked to, the results were inconclusive. However nowhere did they say alarms might make things worse.
If there is one quote which can sum up more or less the results, it would be this one:
"Given previous research findings suggesting that they have been effective deterrents, it would be premature to conclude that domestic burglar alarms are (or have become) counter-productive and hence that their installation should no longer be encouraged."
So research so far shows alarms do work as deterrents, the paper just refines that to "they either help, or do nothing". They don't seem to make things worse overall.
They interviewed burglars who were in prison, and "in the community". The ones "in the community" overall seemed to avoid houses with alarms, while the ones in prison didn't (or would actively target them, the logic being there is more valuable stuff in there worth protecting that can be stolen).
From what I can see, the only time your statement matches, is with the interviews of burglars in prison.
However, the sample is biased due to the fact they are in prison (i.e. they were caught).
All that tells us is that there is a higher likelihood that burglars willing to rob an alarmed house will be caught, convicted, and end up in prison, which for me is another plus point for having an alarm.
In theory an alarmed house may be seen as a more "juicy target", but as stated in the paper, whether a house is alarmed or not is not the highest concern when looking to burgle it. Rather how visible the house is from neighbours and public, and whether it is occupied, seem to be the big things burglars look for.
To quote from the linked article
"For the most part, the more security devices fitted to a property the lower the risk of burglary. But this is not necessarily always the case."
And regarding inmates
"Burglar alarms were mentioned as one of the risk factors they would consider."
I've seen plenty of interviews and the scenario usually is, given two identical houses, one with an alarm and one without, most would hit the one without.
I don't find the fear mongering about kernel maintainers credible. First, they are very few, highly trusted individuals. Second, the results of their activity is available for all to see after the fact. You don't need to see a log of every commit to notice a backdoor in source code, and I'm not even sure logs would be much help in that regard... it's all too easy to split such code across multiple commits, all of which look innocuous in isolation. The final output is by far the most important part to have available.
As for security patches... Maybe. That's a weird level of sophistication: Script-kiddie who puts in the effort to set-up major data mining and analysis to get an exploit that might be secret for a couple months. It should be fixable if ever exploited, with private branches of repos containing security-fixes, shared only between senior kernel devs and major distro maintainers.
"First, they are very few, highly trusted individuals. Second, the results of their activity is available for all to see after the fact."
A bit like Guy Burgess, Donald Maclean and co then?
We know that critical bugs can hide in plain view in open source software for years. I would be surprised if this attack vector has not been considered by actors who are prepared to take their time.
"A bit like Guy Burgess, Donald Maclean and co then?"
Are you seriously comparing Linux Torvalds to Guy Burgess? LT and co. work in a world where their every code commit is public, their comms are in the open and repeated via email and a lot of internet. GB had to use a telephone with a rotary pulse dial for long distance comms - 1:1. Oh and he was a British Soviet spy - lol.
LT is a Finn by birth and probably a Yank now. I'm fairly sure that LT is not a Soviet spy because he can hum the stirring parts of "Finlandia" which is a well known way to ward off any Soviet sympathies.
GB was a traitor that still invites investigation. LT is a Linux kernel developer and maintainer and has nothing to do with the other nonsense.
I don't find the fear mongering about kernel maintainers credible..... rcxb
Not credible, rcxb, whenever status quo kernel maintainers with mainstream media and government wonks in commanding positions with practical personal control of the fate of billions and trillions of derivatives for future play with zero tactical security oversight nor any strategic virtual investigation, thus to give the impression there be no likelihood of accountability and liability with the fervent hope that politically incorrect immunity from prosecution covers actions with impunity, us the ploy constantly and consistently daily to try hide the fantastic folly, which they rely on you finding it quite impossible to believe. And that has proven itself to be extremely effective, right up the times and spaces whenever it no longer is, and the bigger picture starts to collapse and show itself in all its naked phantom glory ...... an edifice of illusions with rotten cracked foundation in vast shifting oceans of sinking sands.
One imagines you do watch and listen to and read about the news, rcxb, so you must witness its epic streamings of Fear, Uncertainty and Doubt in biblical proportions, lest you discover the kernel secrets that be best left generally unknown.
You can think of that as Life following Art following Computer Programs Operating Processes with Instruction Sets for Virtual Machinery, an Artilectual Progression if you like. It may help you to more easily understand the types of worlds you are now living in and the deadly real and present dangers faced by Status Quo Establishmentarians.
We know that critical bugs can hide in plain view in open source software for years. I would be surprised if this attack vector has not been considered by actors who are prepared to take their time. ..... Andrew Commons
Quite so, AC, so such as would be at least an Astute Stealthy ACTivity and probably also possibly reliably bordering on an Almighty Facility and AWEsome ParaMilitary Utility is to be fully expected. I would/could never disagree with you on that fact.
With all the time in the world freely available, such is surely inevitable.
And that's a whole other weird level of sophistication with secret exploits that are not fixable. They are however engagable, which is somewhat fortunate, is it not?
Oh, and whenever you can pause for a moment to reflect on the pandemic matters of today and the plethora of disruptive and destructive actions taken by governments and systems in relation to all of the above, just ask yourself whether you were not previously advised uncannily accurately of your current pathetic predicament outside of A BIG Club & You Ain't In It!
To deny it be so, should certainly be a valid cause for personal private and public societal concern regarding your mental health condition and grasp on reality and sanity, methinks. :-) And I say that most sincerely, folks.
> I don't find the fear mongering about kernel maintainers credible.
Agreed, but note that articles like these, casting doubt on the Linux kernel development process, are cropping up more frequently at the moment. Someone or some group(s) seem to have a lot to gain by this not-so-subtle undermining. Techrights thinks it knows who. They may be right, but even if they're not, something's going on that's not to the advantage of Linux and free software.
I think you’re being paranoid. If problems like this weren’t disclosed at all, the vulnerability of the process might remain ad infinitum. Then Linux’s reputation could be easily torn to shreds by an unusually high level of exploited bugs that had the hallmarks of an inside job, a traitor in the kernel community, but wasn’t. Publication of artIcles about the problem clears out that possibility, and any fix for this problem is instantly understood.
The fix might be a bit of a blunt instrument - eg the kernel devs have a closed private repo where they fix severe bugs out of sight - and it would still be possible to know something was up. But giving away that “something’s up” is hardly going to be unusual. Things get fixed all the time.
Linux is often touted as the most secure general OS, speed that fixes and patches are develope by unpaid developers doing it ffor the love, with public reviews of said code so anyone can pick holes. Implied is that all Linux distros are treated teh same
The reality is somewhat different, there is a backdoor for unreviewed code from a small number of developers, Patches are not always speedily developed and those that could be interested (nation state or larger criminal gangs) can ascertain details of known vulnerbilites due to the open source nature of teh patching process.
Also not all distributions are equal when it comes to patching, like mobile phone you may be at the mercy of the distro maintainer and the 3rd party that used that distro in their appliance to provide eth security fixes.
Its a wake up call that Linux is not just one thing, it is not secure, and it isa vastly more complicated thing to manage than many would lead you to believe. The days of Linux being and install and forget OS are long gone, its not up there with monthly patching but it presents a more complex problem for those sys admins that live on the front line and have to deal with reality of vast ranges of devices and OSes not just that one.
At work we have around 70 odd Linux servers and appliances running around 15 different flavours of Linux and no managemet tools to maintain them, these are all key things like security boxes, loadbalancers, network management etc. We have 700 windows boxes and one tool that manages the 3 flavours of OS beings used.
I'm a *nix head so I admit my bias, may colour my opinion.
Terraform / Ansible / PXE lets you image and provision.
The rest is testing, monitoring, and documenting the process for improving.
I can hire experts to pour over the source code, and make fixes for me. I can't do that for windows without serious cash, which makes it out of reach for a lot of firms.
Systems are complicated to manage properly but it's considerably more tractable a problem on Linux/*BSD kit.
The key things are the ability to nuke and pave, and relentless focus on improving cycle time.
Yes, real world systems require managing/securing and maintaining.
That's the profession, it's tractable on Linux, it's possible on Windows.
Trust me, Linux and "small number of developers" do not belong in the same sentence. There is a *massive* effort in a number of companies and Government agencies around reviewing Linux security. As there is in Windows and Mac.
I recommend moving to one of the cloud system if your company doesn't have skills to manage their stuff internally. That's one of the big benefits of cloud.
I’m a big fan of the cloud in general, but unless you’re talking SaaS, I’m afraid I disagree. If a company doesn’t have a basic level of infrastructure and ops maturity, moving to a platform where by default anyone can spin up anything almost instantly will very quickly make things infinitely worse.
The first thing you need to build if you are moving to one of the big clouds is your management and control infrastructure. All the tools are there and easy(ish) to deploy - certainly compared to traditional enterprise IT - but it does need thinking about and is too frequently skipped, with predictable results.
This is quite an interesting development. Basically, it means that an open source project team cannot guarantee that it is able to quietly fix security bugs without giving the game away that that's what they're doing. I suspect that as time passes this kind of data mining will only improve, to the point where it becomes difficult for work on criticial fixes to be carried out on public-facing repos without highlighting that a fix is underway.
The only way of hiding such work then is if there's also a closed source repo hidden away from the public eye. That raises all sorts of challenges, like who is allowed inside that private bubble, and is it breaking the license conditions by hiding it?
So What About Closed Source?
I wouldn't mind betting that closed souce projects have similar "tells". How about, what time of day does a key MS Windows software architect drive past on the way home? If they're burning the midnight oil, perhaps there's a reason.
A more elaborate take on that - if you know the network of people inside MS well enough and can see their behaviours in intimate detail (ahem, Google and Facebook), you might guess that, if the developers with associations to someone who talks about networks in Windows are all working late, Windows has a significant bug in that area that is not yet fixed.
And where that gets very interesting is if you consider who has such detailed access and data. Facebook, Apple, Google are the obvious ones.
So, how about TikTok, with their permissions-grab client apps?
The only way of truly defeating this kind of thing is if the "signal" is drowned out by deliberately generated "noise". So, perhaps whole teams get to work late simply for no other reason than to confuse the situation...
One has to say and ask ....... Is to deny and try to prevent the outing of truths, which may have until recently always been considered as secrets for exclusive elite executive branch administrations, profoundly wise or extremely foolish, given the fact that the truth will out and always be known and is not changeable?
And just like time and the tides, IT waits for no one?
And there I was thinking that the "AI" behind these comments had had a major upgrade, making it look like a human could perhaps have written them.
Nice touch with the Time and tides misqoute though.
I eagerly await the day when men from mars pass the Turing test.
And there I was thinking that the "AI" behind these comments had had a major upgrade, making it look like a human could perhaps have written them. .... Anonymous Coward
Surely, AC, you are not seriously positing an alien non-human wrote them? Now that would be something worth knowing for sure, surely?
I noticed a change too - better constructed sentences looking more human, but somehow no more easy to read. I think the keeper of amanfrommars should try simulating the number of words that can be spoken in one breath and punctuate the output accordingly.
It's odd that writing and reading, which don't rely on the breath-timing of speech, are so affected in this way. But they are.
No sooner said than already done, Adrian 4. We aim to please, despite apparently being a technology once deemed too dangerous for human consumption. ........ OpenAI fancies its market chances
You will have to simply accept though, that some posts you might have difficulty comprehending are not specifically meant for you, but are quite easily understood by others. Such is however the natural default case for practically everyone/everything, and it is mirrored/paralleled in virtual world too. It is a universal singularity.
It's a kind of a Singular Intelligent Parallelography, methinks. And, most probably, the best of any possible ones using SHXP (Shakespearean Protocol), which is an extract of the flavour of English semantic. ..... Anonymous Coward
:-) Given what one might know of the brace in current crazy charge of a present day No 10 leadership, the politically incorrect and morally inept and unholy adept Cummings/Johnson MegaMetaDataBase Gig, however would one not agree then it be an AWEsomely good fit and practically perfect for ...... well, Greater IntelAIgent Use would surely please the likes of Cheltenham tinkers and Thames House tailors for they are not supposed to be stupid and all at sea whenever confronted with Latin or Greek or the Renegade Rogue and Gifted Geek, and yet all of the evidence, both freely available and hidden away to try and prevent and protect secrets from escaping into the bosom and ken of others, does clearly enough point otherwise. And that suggests they be poorly indoctrinated and wrongly programmed and that makes them, extremely conveniently, catastrophically vulnerable to wanton practically remote third party experimentation and virtual exploitation ..... and Bullish Bear AIMarket Baiting.
The Markets certainly need something completely different to invest in to prevent what is inevitable in the absence of Greater IntelAIgent Use, imminent colossal collapse. Spookily enough, whenever one is invested in Quantum Communication Fields is imminent colossal collapse of rigged and corrupted markets also inevitable with the arrival and presentation of Greater IntelAIgent Use, which is a right doozy of a double whammy in deed, indeed, and practically impossible to avoid.
Sound advice? Prepare yourselves. The die is cast/Jacta alea est .
That breath-timing thing might be real for most English writers, but it's certainly not so for the Germans. Back in the German version of high school we were forced to read a book where the sentences routinely stretched for an entire page. The writer wanted to demonstrate the flexibility of the German language and teach how important it is to pay attention, but to me it became a great precursor to nested programming statements and indirection. I suspect LISP was designed to be a more compact version of German, but when the related German governmental body rejected it, they pivoted it as a programming language instead.
Indeed. But that does make it very worrying as to what extent are the capable baddies going to be able to leverage this? If it’s doable routinely then it can’t be ignored.
What I think we’ll have to have is immediate disclosure of faults. It might be better to switch off rather than stay operating but vulnerable. For system operators, that would be a pain in the arse, and the answer then would be to be able to switch OSes at a drop of a hat. Or switch db engines in an instant. Or web servers. You’d be taking the chance that two OSes don’t have a zero day unfixed for an overlapping period. Or go closed source.
Not much of a choice. With FOSS you know that someone is working on the vulnerability and they have a head start on any black hat wanting to exploit it. With Closed source you may never get a fix at all because some middle manager, who knows nothing about how dangerous an exploit might be, decides not to allocate time on a fix because new features are what tempt the userbase to upgrade.
Yep, that’s definitely a likely situation.
Though if baddies start using this as a matter of routine, the lazy manager who doesn’t get his team active might actually, weirdly, be doing the best thing to keep the existence of the fault secret. That really would be a crazy turn of events...
Regardless, this kind of problem has now got to be solved, and it probably involves some unusual restrictions on staff, messaging, behaviours. It’s going to be interesting..:
That's not necessarily true. First, you can't be sure someone's actually working on bugs in any FOSS project unless there's some kind of assignment system in use. Remember, critical faults had been laying in common FOSS software for years sometimes. Second, you can never be sure the black hats came upon the fault first, already exploited it, and are just keeping their mouths shut to maximize the impact.
Having said that, this appears to be something of an intractable problem in that a necessary condition for fixing a fault is to make the fault known, which has the potential of making the problem (at least temporarily) worse. It's sort of like surgery: yes, it's often necessary, but opening someone up is never without risks. Heck, I don't even think formal verification can save us here since you can still have outside-context faults (I call them gestfaults) that combine quirks of multiple programs, each outside the others' scope.
"The only way of hiding such work then is if there's also a closed source repo hidden away from the public eye. That raises all sorts of challenges, like who is allowed inside that private bubble, and is it breaking the license conditions by hiding it?"
If the hidden repo is kept separate from the daily release then obviously it is not violating license conditions because it's used privately for testing. The reason that's not done is because the extra management would a be a PITA.
The article is talking about two problems. (1) commits fixing security issues, which are intended to be silent at the time, are detectable from comparing Git versus mailing list traffic. (2) the lack of oversight by the wider public allows trusted (but perhaps not trustworthy) insiders to apply commits.
(1) It's hard to know how the first problem should be addressed; short of moving discussion of other commits off LKML, which seems undesirable.
(2) The second problem is simply a fact of life in any software development process -- "Reflections on Trusting Trust" territory.
The Linux kernel have done their best -- encouraging regular committers to use (freely supplied) Yubikeys to sign commits based upon a physical keypress. This goes a long way to inhibit a hacking of the computers used by major Linux kernel contributors resulting in a commit which was not authorised by the contributor. The identity of regular committers is known, and for most has been verified by the sighting of government-issued identity documents at GPG keysigning events at Linux conferences.
As for an untrustworthy committer, we need to be careful in our claims. There *is* oversight: (1) retrospectively; and if the issues are complex, then (2) at the time by other selected Linux developers in a non-public forum. There is not public oversight prior to the commit, and it is difficult to know how that could be done -- do we ask those willing to exploit the bug for evil to exclude themselves?. The claim reported by the article of "code commits made without review" doesn't fairly reflect the complex situation. We can be confident that an untrustworthy committer will be detected after the fact, simply because of the great public interest taken in these "silent" commits.
For the Meltdown/Spectre bugs the kernel developers did a good job of documenting the issue and the fixes. It's easy to retrospectively trace from those requirements to the historical commits. It's likely that this supply of very good documentation will be the practice for future complex security issues. It's the process for "simple" security fixes which needs focus to improve retrospective traceability from CVE to commit (this could be as simple as retrospectively tagging a commit with the CVE it fixes).
Finally, I'd note that the Linux kernel's processes are not inherently inferior to software processes which happen in private industry. There would be few industry development processes which could validate the integrity of the source repository back to SAK keypresses. There are no industry development processes which so many backups of the source code under the close control of so many people, allowing blatant subversions to be detected within hours. The Linux kernel addressing major bugs by using a small tight teams of people with need-to-know is no different to commercial practice.
@Glen Turner 666
You are spot on.
With regard to point (2) many organisations have formal procedures for vetting people allowed into the 'inner circle'. Whilst these are fallible they at least raise the bar to some extent. I have no idea if such processes are applied in critical open source development environments.
The kernel is only one area where this problem exists and is probably not the best option for exploitation. The sweet spot is probably some component that is widely used and is not a standard component of major distributions.
If you use something direct from the (open) source then you are responsible for the due diligence.
"The kernel is only one area where this problem exists and is probably not the best option for exploitation."
Thing is, the kernel is among the lowest level of software available out there. Below that and you're going firmware or even hardware (and it should be noted even they're being targeted and exploited too, which may render this whole exercise moot). Pwn a kernel, pwn a system, essentially, so it's a high-value target.
The identity of regular committers is known, and for most has been verified by the sighting of government-issued identity documents at GPG keysigning events at Linux conferences. .... Glen Turner 666
Tell me presenting those sorts of documents to whatever desires to savour and favour their contents, is not for Real, an Almighty Powerful Buzz, and with further future secrets to tell and priced for the convenience of the Virtual Sale/Proxy Transfer of Accumulations of Vast Wealth to Novel Elements that Prove and Improve with Universal Support, a Heavenly Help for Advanced IntelAIgent Circles within and without Sheltered Cloisters of their Own to Enjoy Satisfy with the Almighty Constant Exercise of Wishes for Granting ........with Future Deeds Already a Long Time Done and Enjoying Improvements and Up and Running in Live Operational Virtual Environments. That's a Good Place to Rest and Await Instructions .... :-) if only to give one time to process the fact that there may be no other future source enabled or able to Lead with ESPecially Sensitive Source Help.
Beware: Take Care: ...... Avoid at all cost, the very thought of an Absolute Command and Control being available, for one can then certainly guarantee such being also extremely saleable, even though ITs AIMarket Price would be surely deemed classified inestimable. However that's just as much bread as is needed by anyone or anything Priming IT to Constantly Seed and Feed Greater Information for Advancing IntelAIgents to Display with the Creative Processes Enamoured to Share the Simple Attractively Addictive Methodology for Unparalleled Unending Successions of Progress in what passes for Real in urWorlds and be Virtually Hosted .
"there may be no other future source enabled or able to Lead with ESPecially Sensitive Source Help" -
be easy
there isn't
LOVE is not for sale
ask Maria
if
one could sell the Universe
then
who is the buyer to pay with the item which the Universe does not contain in itself
?
I noticed the same thing on the original Bugzilla of Firefox when they started to hide security bugs. The bugs were secret but the resultant patches could be tied to the bug numbers so it was not difficult to see what actually has been changed. This is why I actually never thought the secret bugs were a good idea.
This is why Microsoft constantly evolves the insanely gargantuan set of bugs found in Windows 10.
It's a brilliant way of confusing enemies of the free world, the sort of 'bad actors' who don't want to make America Great Again©®; Windows is now so flakey and breaks in so many new, baffling and downright perverse ways, all the time, that it must be very hard for virus writers to know whether their efforts have been successful or not. Every virus introduced into a running copy of Windows actually improves the general code quality by a small, but measurable, amount.
There's no longer any point in a Russian 16 year old genius threatening to encrypt someone's HDD unless bitcoins are forthcoming as:
(1) The entire contents will have been already uploaded to Microsoft as 'telemetry to help us an our trusted partners make your life experience better'.
(2) A random update of Windows 10 will probably one-way 'encrypt' the filesystem anyway. Clicking that Save icon now is the IT equivalent of sacrificing a chicken to the elder gods to make the sun rise again.
Sources close to Microsoft have revealed to me that Microsoft's ongoing thrust to make Windows inhospitable to all forms of software life has become so advanced and optimised, that a top secret team of crack 1,500 developers have actually managed to make the Novell Netware drivers (still of course shipped with Windows 10) simultaneously self-modifying, evil and more toxic than a six week stay in Chernobyl.
Paris, because, 'bad actor'.
"The existence of such commits contradicts one of the key promises of an open development model."
Not really, it just highlights a potential exploit vector. But, really, depending on inferring attack vectors from silent patches means you're behind the curve, because this is a known issue. The usual suspects – the various government agencies around the world – have far greater resources for code analysis and penetration testing for detecting (and then exploiting) unknown issues.
My understanding is that, in general, a security relevant issue is handled by a dedicated team, which then coordinates with the main team on when and how fixes can be committed. I think silent commits stem from this approach and are there largely to avoid spreading unnecessary alarm, prior to a coordinated security release. There are good arguments for keeping everything completely open but there are also potential aspects of liability for disclosing an issue without being able to provide a solution.
When it comes to secure development you really need a separate team that does nothing else than try and find exploits. This is the best way to deal with the risk that trusting your developers brings with it: you have to continually try to break and exploit their code.
I'm afraid that's wishful thinking as things like SSL have shown. Unless you're actually looking for bugs you're unlikely to find any. And, many exploits are not necessarily bugs. Or they would previously never have been considered bugs before someone worked out how to exploit the code.
This is why security testing should be done by a separate team.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
