The unanswered question at CentOS community Q&A: How can we trust you now? The CentOS board conducted a public Q&A just ahead of last week's FOSDEM 2021 open source conference – and there was an awkward silence when someone asked whether changing the end-of-life (EOL) date for a released project is something that might happen again. This killer query came about half an hour into the session, which …
Hmmm...None of them look very Geralty to me. I suppose you mean the wimp at top right, but he isn't very butch, is he, and no scars. The top left guy is surely from the Beverley Hillbillies, or Duck Dynasty, or perhaps from some horror movie where hapless folk wander into some ramshackle homestead in the middle of nowhere, where machetes and chain saws await, since he does look a bit sinister.
Come on, people, they're just standard unix wizards: long hair and/or beards (and mostly white and male). I'm sure few of us are ultimate god- or goddess-like beauties ourselves, and aren't we all supposed to be mature enough to accept everyone for how they look?
Don't get me wrong, the wizard stereotype is a slightly amusing one (and, as keepers of the spellbooks, why not), but I'm not going to criticise them for it.
"...board members assured us that they were "very excited" about Stream and did not show any expressions of regret or disappointment,"
Yeah, they would say that wouldn't they. Keep in with the bosses, after all there are jobs at stake and for that I don't blame them, but to gush all over the place how this is the best thing since sliced bread does come across as a bit forced.
"...in effect forcing the CentOS board to go along with the plans."
Resistance is futile.
Now where have we heard that before?
Many years ago I worked as a professional services consultant for red hat on their JBoss suite of products. There was an internal tension in Red Hat between those who saw open-source JBoss (and later wildfly) as a source of leads and sales opportunities, vs those who saw it as a competitor taking away potential paying customers. Every few years one side or the other would get an advantage and Red Hat lurched from trying to keep important features and fixes out of the opensource stream as long as possible to putting everything in opensource first. It seems that dynamic continues to this day, but with IBM at the helm they have swung further than usual. They will probably swing back the other way in a few years, once they notice the lack of CentOS users impacting their pipeline of CentOs to RHEL conversions.
Interesting insight into corporate dynamics. I've always worked for small/medium sized public sector organisations. I see plenty of churn as a result in shifts in policy from government. I suppose this must be the commercial version, self-generated churn.
"A beta version of AlmaLinux was released on 1 February. "The folk that are doing RHEL 8 rebuilds to keep 8 alive, we wish them the best. We love our open-source friends," said Riehecky."
Pat Riehecky works for FermiLab (was a Scientific Linux dev) and has a significant stake in a stable rebuild OR a 'public service' RHEL licence. I'll be watching the FermiLab news page for any updates... I'm sure he was really amused at this development after the winding down of Scientific Linux and the adoption of CentOS starting with CentOS Linux 8.
Icon: I increasingly resemble Jonny Hughes (top left on the video grab) as I stay locked down...
Redhat has managed to continue to alienate the science community which is where most of Centos was coming from
Their target market is things like banks, with thousands of identical almost thin-client setups and it's getting _very_ difficult to keep science softwrae running on it anymore (which is a real problem as most solar, particle and planetary physics space groups are still using it)
I've been suggesting that our techies learn and support Ubuntu for a while but it's becoming clearer and clearer this is no longer "a good idea" but moving into "mandatory" territory (You'd be surprised how many supposed tech types through up their hands and use the "I don't understand this, it's too haaard!" card - or perhaps not)
> This killer query came about half an hour into the session, which can be viewed here. "A question has come in about the change of the EOL for a community deliverable during a release being very unusual. Is this [a] thing that could theoretically happen in the future?"
> ....
> A lengthy silence ensued before Pat Riehecky from Fermi National Accelerator Laboratory, a CentOS board member since April 2020, offered: "It's hard to predict the future."
Presumably the silence was everyone hoping one of the others was going to take that bullet.
The answer, clearly, is - yes it absolutely could happen, just as all the ground they're "giving" on licensing to ease the transition to Stream could evapourate.
Essentially, you need to either be thinking of changing distribution entirely, moving to one of the new community builds, or preparing for the costs of entering the RHEL licensing ecosystem.
"or preparing for the costs of entering the RHEL licensing ecosystem."
Those costs aren't JUST the licensing. It's RHEL being being seriously crufty in a bunch of areas and the RHEL rpm environment not being something you can just drop XYZ pakaging RPM into
EG: Just because SSH in RHEL7 says its 7.0.2 doesn't MEAN it's 7.0.2, it may incorporate chunks of later versions (including pieces of SSH 8) AND still be crippled by pieces of SSH 6 as well. (I'm bringing this one up as it's a real world example and it's major security issue. The only way to disable dangerous ciphers is to hardcode the exact list of what you'll accept, which in turn means you need to look at every update to see if things have been changed/new ciphers added
Of course researchers being researchers, they'll see that XYZ package isn't the latest and greatest, so compile up their own version and locally install it, leading to all kinds of library clashes and the inability to keep rpms portable across systems unless extensively customised - which defeats the purpose of using a distribution-based system in the first place
You can tell people not to install latest "everything" and build against that until you're blue in the face. It doesn't sinnk in until the $2million contract you have with ESA is being cocked up because such decisions mean the software you supplied them won't work in their standardised environments (yes, really).
It's not just the software / OS world that is built on trust; the best vendors in technology realise this.
I had $BIG_VENDOR decide to EOL a bunch of products with no possible last time buy. They got moved to the 'Do Not Use' category really quickly.
In high reliability and other areas that need true long term support (think decades) the trust in the vendors to continue to support specific parts becomes a major part of the decision making process at the design stage.
If a part (especially in some areas) becomes EOL, it may be necessary to re-qualify the equipment (and even partial requalification can be $$$$$$$).
A vendor who has lost the trust of any of the larger companies in those areas (and others) will slowly but surely lose design wins to companies who actually keep their word. Word does indeed get around - it is not that huge an industry and many of the people in it are pretty connected even if they are in competition.
Design win on a product with a long lifetime requirement == long term sales. Keep your word and design wins++. Rinse and repeat.
Note that the vendors do not have a contractual arrangement to supply parts for that amount of time (unless you want to use a die banking service - $$$$) but they are acutely aware that the trust of the developers is their single biggest asset.
Lose that trust and those designers will never use your products again wherever they happen to be working.
If you ask me, even 3rd parties need to watch out.
Now if the 3rd parties certified non-RHEL already, then they'll certify non-RHEL again, just somewhere else like Debian. This move by IBM kills off a lot of software and trickles all the way down to the desktop. Enterprise software is glaringly obvious, but also for instance software like Davinci Resolve just lost a reliable non-commercial host.
The code battle is ironic a bit and worriesome further. While granted not identical, consider SCO vs. IBM. Now IBM vs. CentOS? Can IBM see a future of claims against CentOS forks similar to SCO's claims? I know that sounds crazy but, so did the SCO vs. IBM and that still happened.
There's still openSUSE Leap, after all SUSE is the #2 enterprise Linux vendor after RH and most ISV applications that are certified for RHEL and CentOS are also certified for SUSE (for example, SUSE Enterprise Linux is the primary platform for SAP HANA).
If you have to get support then SEL will save you some bucks over RHEL support, and at least for us SEL support has been excellent.
@Smirnov
I know little about the SUSE world. Is it meaningful for an ISA to certify software on OpenSUSE for intended use on SEL? Or does it work the other way round (software certified on SEL will work OpenSUSE guaranteed?)
Icon: I'm a clueless end user, just interested
" after all SUSE is the #2 enterprise Linux vendor after RH "
Suse have proven themselves quite willing to toss non-german speaking customers under a bus and run away (not even responding to quesries on the issue from Novell head office)
We're talking serious and permanent breaches of trust. They're on my "Never do business with this company again" list
No, it can't, and that's not just because of the lack of ISV certificates. There are other issues, for example the fact that Debian is maintained by a community with a long track record of focusing more on empty activism than to improve their software quality. Part of the problem is also that many contributions come from people who don't know how to write proper code or even a bug report, and that shows. The ridiculous OpenSSL fuckup was just one of many miss-steps of Team Debian.
Then there's the "support" cycle which is just 2 years through the Debian maintainers and an additional three years through a completely separate (and much smaller) group. Which gives you 5 years of support through volunteers. Which, again, means the quality varies a lot and fixes can take a long time to ripple through (the fact that Debian sticks with mostly very old package versions doesn't help here, too, and there's not much backporting in the way RH does).
Debian certainly has its place, and might be a good starting point if you want to create your own distro or hosting platform from scratch, or are into fiddling in the guts of open source. Not for a business for which the OS is merely a tool to run specific applications.
"The ridiculous OpenSSL fuckup was just one of many miss-steps of Team Debian."
Anything more recent than 2006 that is specific to Debian?
Icon: I'm a clueless end user, and I'm glad you are happy with Suse for your customers/users
Plus the CentOS board have no power to make any decisions about it at all so their "It's hard to predict the future" really translates as "we do what we're told".
The majority of CentOS board members are Red Hat employees and most of them, if not all of them, are nowhere near the C-level execs needed to make such decisions. Some others are not RH employees but when the vote goes 7:2 (or whatever) it's not hard to know who voted where...
I used CentOS Stream before it was "the focus" and I think people might be freaking out a bit here and misinterpreting what it is. It's patches which form a gradual set of changes between minor releases. Minor point releases are like service packs when it comes to server components. Contrary to what folks say, it's not a development system like Fedora or a rolling release system at all. If anything, it gets your real world bugs fixed quicker without breaking ABI compatibility for components which are marked as ABI-stable.
Put another way:
* How many people running servers would delay switching from say 7.1 to 7.2 and risk compromise?
* How many did NOT have their yum config simply specify 7 rather than a point release anyway?
* How many people are testing their patches in a dev environment, then staging them for production?
If you don't do the first two but follow the best practice of testing patches anyway, then Stream is less work to maintain compared with point releases anyway because changes are smaller, rather than in one big bang, getting your real world problems fixed quicker when you report them, despite not being a paying customer.
The only down side I can see is that you have to migrate to the next major Stream every 5 years rather than every 10. However, with the rate of change Linux undergoes, I'm not so sure that's a bad thing these days. CentOS 6 was hideously broken on newer hardware towards the end of its lifecycle and 7 is already getting that way.
The downside that you are missing is that with the old style CentOS, you could plan for a point release coming along and including new things that would break your system. So you knew when to watch out for breaking changes and could plan for them. In the new scheme of things you will now get breaking changes whenever Red Hat feel like pushing them.
Its not just uncomfortable, its a downright insane place for a production environment. No one who values their clients or their job is going to use it in production because its only a matter of time that some untested piece of beta software breaks something very important.
Fundamentally, CentOS stream is a beta where Fedora is an Alpha. It is never a good idea to use beta software in production anywhere outside of a UAT environment with power users, QA testers and developers that know they're testing something and that it might break. That's different from an unsuspecting client who expects the software they're using in production is stable and probably won't unexpectedly break just because the vendor pushed out something that broke a mission critical piece of software.
What a silly comment.
Especially:
"How many people running servers would delay switching from say 7.1 to 7.2 and risk compromise?"
If there's a known security vulnerability, it gets patched and backported pretty quickly. If there's a bug, those also get backported. The idea is your software version stays the same and you can PLAN for the bigger updates. You obviously have no idea how to run servers in a production environment.
Non-current minor releases don’t get security updates backported because CentOS doesn’t include EUS errata. Without EUS, holding off on a minor point release means no more errata whatsoever. That’s why people have their repos set to 7 and not fixed to a given point release, so they get the latest patches when they are available, precisely because patches are NOT backported to non-current point releases on CentOS in the default Base repositories.
So let me reiterate: How many people running servers would delay switching from say 7.1 to 7.2 and risk compromise?
The answer to this is that when some vast half-working system is finally persuaded to actually work, when arranging downtime takes hundreds or thousands of hours of people's time and when outages cost them business or, worse, end up on The Register (or, worse, the TV) people become very, very frightened about deploying upgrades indeed. And when the existing system is deployed on x.y then you very definitely want and new OS instances to also be at x.y, not x.z.
Yes, many of these things are bad practices, but they are also what life is actually like for many organisations. The flip side, of course, is that these people are already on RHEL, not CentOS. The people who were doing this on CentOS were probably alway going to get screwed, and they just have.
If RedHat's already got a Fedora Server, then introduces a CentOS Stream to fill exactly the same niche, won't they run the risk of confusing their customers? Confusing customers - what a way to go - they usually go elsewhere. Anyway, they've sold out one highly valuable brand and I doubt they'll be getting its equivalent back any time soon.
On the upside, AlmaLinux beta's usable. I haven't tested Rocky Linux but it still hasn't graced the download links yet.
They're not saying this, but the way I'm looking at it is that Fedora Server fills that traditional role that an alpha did back when the software lifecycle made sense, and that CentOS Stream fills the beta role. They're both nothing anyone with sense is ever going to use in production.
IBMhat thinks that they're going to drive people purchasing RHEL by doing this, but its going to alienate people who were using CentOS as a relatively stable environment to try to push people to buy RHEL by showing how well it worked
CentOS 8.0 Linux, built from the same sources as Red Hat Enterprise Linux (RHEL) 8, was released on 24 September 2019 with a projected EOL of May 2029.
In December 2020, community manager Richard Bowen posted about the "shifting focus" to CentOS Stream, announcing that "CentOS Linux 8, as a rebuild of RHEL 8, will end at the end of 2021."
Oh wow, somehow I missed that they were doing this.
Red Hat just got themselves added to my "company cannot be trusted, do not use" list, right next to Microsoft.
If I go back on my word, then I'm a prick who should not be trusted. Regardless of whether we have signed a contract or not.
All signing a contract does to change the situation is that it gives you some recourse if I do turn out to be untrustworthy.
But the lack of a contract doesn't mean that ethics somehow don't apply.
So I'm not sure what point you're trying to make here? That the people who have been misled have no recourse? Or that they were naive for expecting others to act in good faith? Or perhaps you're saying that there's no problem with this massive breach of trust because the law doesn't care about ethical behaviour, so to hell with ethics as a concept and tough shit to everybody who has been misled?
The point I'm making is that people who opt to run mission critical services off the back of a service that is provided for 'free' are not in a strong position to complain if the ground shifts under their feet.
'Promises' are easily made and easily broken, for all sorts of reasons - good, bad and through force majeur.
If I'm relying on someone else's goodwill for a service I had better make jolly sure I have a 'Plan B' in place, and am willing to 'pay the price' of free-loading on someone else's goodwill, because in this life things can suddenly change. Whining never impresses.
All true. However, the other poster's point is also valid.
Adopting any open source (especially free-as-in-beer) software requires trust that the provider of that software won't suddenly turn around and shaft you. Cutting off early adopters of CentOS 8 the way they did means Red Hat are no longer worthy of that sort of trust. Red Hat may have acted entirely within their legal rights, but they've hosed their reputation (and possibly lost a source of sales leads) by doing so.
Yeah, technically you're right and your point is valid. But I don't think it's unreasonable for people to be surprised or annoyed in this instance. Most people using it do have an (unwritten) contingency plan for this scenario: switch to another distro. But I see nothing at all wrong with grumbling that you were lied to and so you now have to enact that plan - a bunch of work you weren't planning on doing.
We all know that that planet-killer asteroid is on it's way, but I bet you'll still be surprised if it lands on your house tonight.
And I'm also willing to bet that when it does land on your house, you're not going to think me saying "well you knew mars was right there the whole time and you knew how the dinosaurs died" is very helpful...
Some risks are worth planning for, others you just have to say 'Que sera, sera'.
The killer asteroid strike - shit happens.
The distro maintainers doing a U-turn on a 'promise' they made to... actually to 'no one' because everyone using the distro is doing so of their own choice and at their own risk - I have to assume it's probably going to happen within my time of use, and if not that then some other distro-screwing tragedy. I can wail at the moon when that happens (but no one gives a shit), AND have my Plan B in place.
The distro maintainers doing a U-turn on a 'promise' they made to... actually to 'no one' because everyone using the distro is doing so of their own choice and at their own risk
Redhat did make a promise: they promised to provide updates for 10 years from initial release. They made the promise for both CentOS and RHEL, and reneged.
I think your point is that people and businesses that use only CentOS saved on license and support costs by assuming the risk themselves. If they no longer like that risk, they can start paying for support or find someone else to provide a similar service for free. That seems reasonable.
Some people run CentOS to develop for a target of RHEL. They are not paying Redhat, but their customers/employers are. Some businesses run CentOS where they will self-support, and RHEL where they want outside support. In both of those situations, CentOS is part of the overall value proposition that RHEL offers and for which they are paid.
Taking a known risk is true however one looks at the history around CentOS and as far as I can tell there is no indication that this was going to happen.
It certainly has not been at the top of our list of risks when we rolled out thousands of CentOS nodes in place of Scientific Linux.
You now need to assess the risk they pose to your business and react accordingly.
If your plan runs for 10 years and you don't like paying for a supported OS to host it I'm not sure what your options are. I'd be questioning the business plan if it was me.
"Q?: How can we trust you now?"
A!:You can't because that horse has already bolted. What you can do is use and support one of the community replacements for CentOS such as Rocky Linux.
The moral of this tale is that big tech companies ought to think first before they act and ask questions of themselves, "How will this affect trust in our brand if we do this?".
"The board and Red Hat want Stream to succeed. If we make it terrible, it will not."
Mmhmm. That's certainly true of every product squeezed out of a corporate orifice. No bad leakages have ever managed to stain the drawers of anyone unfortunate enough to get too near. It's all good and logical. Yes.
Exact time of the "question", 34:00: https://youtu.be/CXFrBv1wwvg?t=2040
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
