'Uncertainty' drives LinkedIn to migrate from CentOS to Azure Linux Microsoft's in-house professional networking site is moving to Microsoft's in-house Linux. This could mean that big changes are coming for the former CBL-Mariner distro. Ievgen Priadka's post on the LinkedIn Engineering blog, titled Navigating the transition: adopting Azure Linux as LinkedIn's operating system, is the visible …
Giving them some credit, MS has been doing that for decades. Once they built DOS using Xenix. Windows NT has grown and matured beyond recognition. I dare say that they’ve had to use and fix Office a lot. Their dev tools are pretty good. Under the current management they seem to have been very open to all sorts of things and played relatively nicely.
They could do pretty well if they really put effort into a Linux distribution.
When one considers what companies might put up some stiff competition against RedHat / IBM, MS is probably one of the likely candidates whom may even be acceptable to many in the OSS community.
I’ve sometimes wondered if an MS / Ubuntu merger could ever happen, and what that might produce.
An MS/Ubuntu 'merger' would be more of an acquisition since Canonical is such as small fry compared to Microsoft, purchasing them would be like spending some loose change they found down the sofa to Microsoft.
I'm personally not sure how well it would go down with the OSS community though as some of recent Ubuntu decisions have already caused disliked by some parts of the Linux community. So being owned by Microsoft would probably add to the distrust and result in people migrating away from Ubuntu and onto another LTS distro.
[Author here]
> Once they built DOS using Xenix. Windows NT has grown and matured beyond recognition.
All true.
For me the disappointment is WSL 2. Not because it's not clever: it is clever.
But because WSL *1* was cleverer. It was in essence a complete ground-up implementation of a Linux-compatible OS layer on top of NT. That is non-trivial. Solaris does it, FreeBSD does it, even Plan 9 does it, but it's hard and all are dated and incomplete.
So was WSL1, and the performance wasn't up to snuff.
But I wish it had continued with WSL1, made it a full Linux kernel personality, and integrated Linux filesystem support into the NT kernel so that WSL1 performance got better. That would have been a bigger threat and competitive pressure on Linux, and it needs some better competition.
QNX does it too...
WSL 1 indeed should be the "better" answer. However, there is one particular aspect of WSL 1 that is highly, highly intruiging and vexing, and may be the reason they went to WSL2.
In POSIXes, there is the select() function call (and associated functions like epoll() in Linux). With this, a program can block until a file descriptor becomes ready. And because literally anything is a file descriptor, that means select() can wait on a large variety of interfaces such as tcp/ip, serial ports, actual files, pipes, you name it. On Linux, there's also things like the inotify interface and signalfd too, so select() and its ilk can wait for some truly diverse stuff to become ready, at which point you can decide what to do about it having become ready.
You can't do the same on Windows. Ok, you can for tcp or udp sockets, but that's it. Nothing else. There is no full featured equivalent in Windows at all. When you use methods such as WaitForMultipleObjects(), you're necessarily waiting for a callback to have completed an IO operation; you're not going to get told that the operation can start.
This has had a major impact on some key project. If you dig around inside the developer chat for cygwin from a long time ago, there's the conversation they had when they came to realise that implementing select() and have it work for all manner of input types (such as a cygwin pipe, which was simply mapped to a Windows pipe) was a none-starter. What they ended up doing was for every cygwin file descriptor in a call to select(), they started up a thread for each one that would poll the underlying Windows object for readiness. Very inefficient, very 1980s POSIX (some *nixes back then did the same thing to implement select()). It's also why ZeroMQ does not support ipc (pipes) transports on Windows. It's also why C++'s Boost asio library is proactor with callbacks (as this page makes clear).
Fast forward to WSL 1, and suddenly there on Windows is an implementation of Linux's select() and its related functions. I've not tried myself, but I think it higly unlikely that this select() is anything other than full implementation. I've no idea what that implementation is, but it's there.
Which begs the question, if WSL 1 can do select() properly, how? Has the WinNT kernel been altered to support it? Was there a big engineering job done on the WinNT kernel? Can or will this functionality be exposed in Win32? Or, is it somewhat borked to make it work (cygwin style), and is this why WSL.1 is considered a dead end?
I don't have any answers to those questions. But it is intruiging.
This difference between *nixes and Windows is perhaps the most fundamental difference I can think of, and it's the difference I find most frustrating. There's stuff that you simply cannot do on Windows because of it.
So, I wish that not only had they completed WSL 1, but that they had exposed the same "reactor" interface to Win32 applications (and thence into .NET).
File Systems
Integrating Linux file system support would have been fairly easy. There are commercial drivers for some of them for Windows.
A small pity is that - with probably not too much effort - it could be something that the file system projects themselves created (the device driver SDK for Windows to be able to do this is free). This would be especially easy if someone wrote a basic driver that translated between the calls Windows wants a filesystem driver to support and the functions that Linux file system drivers need to implement. Such a thing could be a universal wrapper around any Linux file system, and so need be written only once.
Is a Linux Personality better than a Linux VM?
There may not be any noticable difference. For example, in Windows 11 with its virtualisation based security, all applications are running inside their own VM. The fact that WSL2 processes also run inside a VM (albeit a shared one) is not such a radically different thing to any ordinary Windows process
Er. I find this comment strangely seems to flow backwards, from conclusion to premise. I literally can't make head or tail of it.
> QNX does it too...
Does what too?
[Lots of tech stuff elided]
> You can't do the same on Windows.
OK, but you see, this is my *point.*
> Is a Linux Personality better than a Linux VM?
> There may not be any noticable difference.
That is of course the desired outcome. The user should not be able to tell any difference.
But why make it a kernel personality? *Because*, as you said:
> You can't do the same on Windows.
If Linux support _were_ a kernel personality _then the NT kernel would have to do that stuff_.
That _is_ the point.
One of the upsides of WSL1 was it gave Microsoft the WINE problem of changing underlying implementation bringing out bugs of software above. But with the GNU/Linux world, that all open, and apparently MS were doing the right thing and fixing stuff and submitting patches. Which is good result. WSL2 gives less to the wider open source ecosystem.
There was a time I thought a Microsoft Desktop Environment atop a Linux OS would be a useful innovation, but MS have tried their level best to mess up the Windows GUI over the past decade or so.
I do still smile at the phrase "Microsoft's in-house Linux", not bad for a plucky OS the CEO once described as "a cancer"
"When one considers what companies might put up some stiff competition against RedHat / IBM, MS is probably one of the likely candidates whom may even be acceptable to many in the OSS community."
I don't know why you assume that, and I might be wrong, but I don't think MS would be accepted. It's pretty ingrained in many free software advocates that Microsoft is the enemy, has always been the enemy, and will always be the enemy. When Microsoft builds in a Linux layer, I see people crying "embrace, extend, extinguish" at it. When Google builds Linux into operating systems, then denies users the ability to do anything with it, I see people trying to convince me that Android should be counted as a Linux system and means that open source is winning. I think that Microsoft could develop many useful tools that work well with Linux, but if they tried to make their own distribution, I expect lots of people to refuse to run it on principle.
That certainly is the kind of thing that does happen, and I can see people / projects / companies indeed being wary.
However, with RedHat / IBM making a power grab in the Linux world I think that there's a possibility that we may all soon be looking for a strong company to resist. Imagine RedHat starting to do with SystemD and Gnome what they already do with their modifications to the Linux kermel.
As the power grab is all about control of developers, an effective fork requires putting together an equivalent dev team who can out-develope RedHat / IBM. There is no one in the Linux world at present doing that, though even Oracle seems to be realising it might have to to keep it's own RedHat inspired distro current. Why not MS?
MS and Oracle teaming up would indeed be interesting, especially if they teamed up with Ubuntu and Suse.
I think a lot of whether such a thing happens or not comes down to how pro-active the companies want to be. There's a growing threat to their Linux-dependent business from RedHat's control of key projects. It's a case of whether or not they lose big time if RedHat does go nuclear on doing the same thing with SystemD and Gnome as they have done with the kernel. If RedHat starts running a private fork of these projects that is dominant due to the exclusive development effort applied to it, that would be bad for all other Linuxes that depend on those projects.
Open collaborations are good for such companies for other reasons: it's very much not anti-competitive, if the result is openly available. Being seen to not be anti-competitve seems like a good idea at the moment. Microsoft seems mostly (barring the odd blemish) to be avoiding anti-competitive investigations; there's no harm armouring one's reputation more against such investigations.
Another interesting thing about a grand collaboration is that it could "exclude" RedHat. By deliberately not permitting RedHat to download binaries from the collaboration, they could also stop RedHat accessing the souce code (at least, for GPL projects). I.e. the grand collaboration could do to RedHat what RedHat is doing with their flavour of kernel (and could also do with SystemD and Gnome). RedHat / IBM likely cannot out-develop a grand collaboration of major companies.
The reason they won't is that's not at all a clean slate. Running atop Linux means bringing in all the things Linux has had to do for decades. That has some advantages, because they don't have to write their own implementations of filesystems or the many other components that Linux has installed, but it also means they can't do any of the things that designing from the ground up generally lets you do. They would be trading the 1990s restrictions that went into the NT kernel, many of them based on older restrictions carried over from VMS, DOS, OS/2, and other operating systems for a different set of 1990s restrictions, many of them based on older restrictions from Unix and BSD. If they want to run anything natively, they'll also have to keep around quite a few of the first set as well.
Designing that way doesn't let them do any of the things that starting from scratch and building an NNT kernel might. Not that I expect them to do that either, but if you're going to go through a long and expensive migration project, doing one that lets you design anew is probably considered more worthwhile than doing one that, if successful, means everyone is at the same place they were before and it's easier for people to stop using the product.
"If you want to improve your company product make your company actually use it in real world conditions"
This makes me wonder about their patch teams. Do they really put up with their own product? Or maybe they do and have no experience of how much better it could be. Perhaps they should be made to run something like Debian as their daily drivers for a couple of calendar months and then switch back to be faced with the accumulated junk they'll have to catch up on.
Time travelling energy supply? Taking energy from a future spectre with an excess of angular momentum? Well given that many companies are all about money now, stuff the future, your description is highly plausible.
Better watch MS carefully. If they come up with both time travel and infinite energy, their stock is going up.
Time travelling energy supply? Taking energy from a future spectre with an excess of angular momentum? Well given that many companies are all about money now, stuff the future, your description is highly plausible.
Sounds like the Weeping Angels' MO.
Better watch MS carefully. If they come up with both time travel and infinite energy, their stock is going up.
Then definitely worth keeping an eye on. Actually... don't blink. :)
I'd love to know more details. Surely it's a website running with a database behind it and some back-end apps. If they're not OS agnostic (even within the Linux sphere) I wonder why? And does that show poor planning on their part? Or is it just a case of something that grew organically and so future-proofing was never "baked in".
It sounds like the major parts were the lack of GUI, requiring a remote development environment to be set up, and the custom(ish) filesystem.
Everything else appears to have been relatively trivial - at least, far easier than moving from one version of Exchange to another.
[Author here]
> Everything else appears to have been relatively trivial
I don't know anything except what that blog post said, but it absolutely does NOT sound trivial to me. It sounds like this was a massive exercise, one that they've probably tried 2-3 times before (as supported by the 2023 story that the Azure migration had failed), and which has resulted in massive changes not just in Azure Linux but also in MS' internal Linux infrastructure.
AzLin now has a new filesystem. That's trivial, yes. It runs systemd in container. That's pretty trivial.
MS now has an internal container image repository and a mechanism for pulling images from it onto AzLin instances. That's quite bit.
Azlin now has signed kernel images with non-free drivers in. Which I'll bet you means signed nVidia drivers. Which means an internal nVidia repo and driver build process. That's not trivial.
Azlin now has infra for deploying onto Secure Boot machine with full signing and verification. I wonder if that prompted the discovery of the SBAT issue that killed a load of dual-boot boxes last week?
https://www.theregister.com/2024/08/21/microsoft_patch_dual_boot/
Wouldn't bet _against_ that.
It sounds to me like LI was running a lot of CentOS 7 and maybe some 8 and with their EOLs getting close, some time a year or two ago, LI and MS worked together to upgrade Azlin to do what LI used from CentOS, and then deployed it... and it was very hard and we can be confident that there was _at least_ one failed effort.
RH forced MS to up its game and bring MS Linux up to snuff, and test that in prod at considerable scale.
It's hilarious really.
I don't consider trivial to be the same as easy or quick.
A long list of trivial things is still a lot of work, it's just not work that requires rare skills or particularly special knowledge.
"Compile this massive codebase" is often trivial (click one button), but may take several hours to actually complete.
"Fix the errors and warnings after updating the compiler" is often trivial, but may take a long time to do.
And trivial fixes often require a lot of nontrivial work to actually find.
Running at that scale is a non-trivial problem. They're running from multiple centres but still have to keep the data in sync.
Then there's risk aversion. Because of that production distros such as Centos tend to be trailing edge in terms of release versions of different components but with security updates back-ported. They're going to have to build and, also because of risk-aversion, thoroughly test all their custom stuff onto what may also be a distro similarly customised for extremely long term support but with a more recent base.
First reaction was WTF why are they just deciding on moving to another distro now(reading the first line of el reg's article), after EOL is past for CentOS 7. But the first line of the blog post says as of April 2024 almost everything is already moved.
Skimming the blog post seems most of their challenges would have been the same regardless of what they were migrating to, as CentOS 7 is a 10 year old platform and I assume most updates during that period were just back ported to their own versions, even going to whatever the latest RHEL is would be a similar jump.
Sounds as if Azure linux is RPM-based. Odd that they felt the need to connect a GUI for their IDE for development. As long as I can recall at least the developers I've worked with going back maybe as far as 2006 or 2007 used Macs for development and that stuff got pushed to linux servers (same holds true for the org I am at now). Not many issues, though once Macs went ARM the number of issues for developers went up a bit (but they handle that themselves). Though all of the languages they use are interpreted,nothing C/C++ etc. I did maintain a developer VM of one of the app stacks a previous org ran for several years, used the same OS/configs(as close as possible) to production. Though that certainly goes out the window with developers using ARM and the servers being x86.
Myself, waiting for Ubuntu 24.01 to come out to start testing transitioning from 20.04, haven't seriously run CentOS since CentOS 5. I was RHEL/CentOS only at work from 2003 till about 2010 then Ubuntu since. (and Linux Mint as well as Devuan at home).
I don't understand the reasoning re XFS being a differentiatior, and Azure adding support XFS.
Why do you need to 'add' a filesystem that is already in the kernel [1]?
The user space tools have been around in open form for a while [2].
[1] https://lwn.net/Articles/838819/
[2] https://wiki.archlinux.org/title/XFS
From the LinkedIn blog post in addition to XFS they also used software raid but didn't specify what (LVM or MD) or why or for what purpose.
All up I am suspicious they were running a frankensystem built from bits of Centos7 and and an eclectic assortment of other components. Even transitioning to RHEL8 or RHEL9 might have been equally problematic.
Wasn't very clear at various points whether the blog was referring to virtual machines or user space containers (à la Docker) or or VMs hosting such containers.
Trying to manage a decade or so of Heath Robinson like lash-ups is never going to be easy to deploy or admin and definitely not on a new platform. ;)
> Wasn't very clear at various points whether the blog was referring to virtual machines or user space containers (à la Docker) or or VMs hosting such containers.
My suspicion is that this is entirely deliberate, and the reason is that the real answer is:
"All of the above, and some more besides."
I've heard it before and I'll hear it again, and I probably won't believe it then either. In my estimation, Windows will stay Windows while people are still buying it and running legacy applications. If enough people stop doing that that it's not worth keeping around, then it will die, and people will use something else and use VMs or compatibility layers for emulating it when necessary. I don't think that will happen soon, as I don't know what is likely to replace it in the near future. A ground-up reimplementation seems unlikely at all, and if it happens, I see no reason why it would use a Unix or Linux base.
[Author here]
> Windows will eventually become a Unix based operating system
I've read that kind of comment in several places in response to this article. I can't offhand offer a better rejoinder than I did on Mastodon:
Actually, no, I strongly disagree.
I didn't think Unix was the most advanced design when I first met it in 1988 & it's much more dated now.
Linux *is* obsolete, as Andy Tanenbaum said >30 years ago. He was right.
Underneath all the marketing cruft, NT is a good kernel and a good OS.
If MS continues to try to "embrace and extend" it with WSL etc., then maybe that will pressurize Linux to up its game and improve.
Monocultures are bad. We should not want less OS diversity. We need *more* of it!
> More advanced != better
No indeed. Real brilliance, real advancement, comes from simplification rather than adding complication.
For instance: eliminating rival vendors' products; reducing the number of products used; using in-house tools and people rather than external resources; and generally simplifying the stack as much as possible. For example by removing multiple distros and replacing them with one distro running containers that reproduce those distros.
With respect, some clarification on this would be valuable. Because honestly, I have never heard anyone say that since a Microsoft Program Manager made the claim to me in 1996 and I laughed then.
Didn't el reg run an article sometime ago saying - along the lines of - the kernel suffered as security was added as an afterthought?
Also, given MS often give their product away for free to trap people into using it... why isn't anyone using it for anything other than staff desktops?
there are alternatives - viable alternatives - if you look at SUSE's Liberty Linux. BUT the interesting part is that Microsoft considers its internal Linux as a strategic investment otherwise they would gladly buy all they need. Plus it is a constant source of how LinkedIn engineering can fight off attempts of Microsoft to impose its procedures and standards on the LinkedIn tech realm ;-)
[Author here]
> BUT the interesting part is that Microsoft considers its internal Linux as a strategic investment otherwise they would gladly buy all they need
No, not at all.
I think the reasoning is more like:
1. Don't give money to commercial rivals.
2. If you need something that comes from a rival, look and see if any of your own divisions has anything even vaguely like it.
3. If you do, USE IT. Then you can keep that effort in-house and improve your broader portfolio.
4. If you do pay rivals then even if indirectly you are endorsing them and their products and denigrating your own.
I know from direct personal experience that of the 3 significant enterprise Linux vendors, 1 runs on Office 365; the second, I am pretty sure now runs on Gmail; and I have heard rumours that the 3rd also now runs on Gmail.
All 3, of course, could do this 100% in house with tools and expertise they own, but management are not smart enough to know that.
The simplistic analysis is:
"Would it be cheaper to do this stuff with 3rd party tools or do it in-house?"
The answer is, yes, it's cheaper to outsource it. But that is the WRONG ANSWER.
The deeper question is:
"CAN we do this in-house and thereby improve our own products and services?"
"If we do and we spend more money, how do we benefit?"
To which the answers are:
"We avoid helping rivals and endorsing their products. We may spend more but we keep it inside our own company, not handing revenue to others. We get more knowledge and expertise in house, and we reduce our dependence on others. We reduce our exposure to external threats, and to external spying."
If your management thinks it's all and only about the money, that means it's bad management.
[Author here]
> Good advertisement for Windows and Dotnet.
No, not really.
Note this is not about WSL or anything. The clue is the stuff about Secure Boot and signed kernels. This is about bare-metal Linux.
As such, it's not an endorsement of Windows or .NET. It's an indictment. It means that after over 20 years, they have not been able to move this to Windows, and they have been obliged to keep it on Linux.
But now, at least, after major effort, most of it is on their own Linux.
It's the direct exact opposite of what you said. It's an endorsement of Linux and an indictment of Windows and .NET.
> Then again doesnt hotmail run on *nix as well ?
Er, no. May I suggest that you read right to the end of the article?
However, the Azure cloud insists on signed kernels, so Microsoft has built a repository of signed kernels incorporating all the hardware in use.
Gosh, I wonder who's responsible for signed kernels being difficult?
Still, double schadenfreude in RH pushing innovation in Linux... by inadvertently forcing MS to do it.
Oh, don't get me wrong. I am not a fan of "Secure Boot" at all.
But I respect efforts to work with it, using kit you already have, especially where this means creating simpler processes and simpler tooling to replace on-the-fly rebuilds.
We should all listen to RMS more, I reckon.
https://www.stallman.org/articles/microsoft-talk.html
But I respect efforts to work with it, using kit you already have, especially where this means creating simpler processes and simpler tooling to replace on-the-fly rebuilds.
They do seem to be improving (or, as it's a massive organisation, some factions appear to be doing things that might be construed as good*). I was mainly enjoying that 2024 MS is having to deal with problems that 2011 MS inflicted on the rest of us.
Of course now nvidia have "open sourced"** their kernel module this may become a moot point.
* for values of "good" that include running linkedin
** for values of "open-sourced" that include "moved 60MB of closed source into userspace"
Is your objection that some software, including some Microsoft software, has decided to install itself into the user's directory instead of Program Files, thereby allowing an unprivileged user to install it? If that's not your complaint, what is the problem you are trying to solve? If it is your complaint, you are aware that people can install binaries into directories they have access to on Linux, that removing them from sudoers doesn't stop them doing it, that you can stop them doing it if you're willing to put heavy restrictions on what they can do, and that Windows has the same ability to take that broad an action?
"They've found a way to connect IDEs running locally to remote Azure Linux VMs, complete with GPUs – over 1,500 of them, in four different geographical regions. As you would expect, it mentions Microsoft's own VS Code ... "
Not much to figure out here, it's trivial, we all do it all the time.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
