back to article The unanswered question at CentOS community Q&A: How can we trust you now?

The CentOS board conducted a public Q&A just ahead of last week's FOSDEM 2021 open source conference – and there was an awkward silence when someone asked whether changing the end-of-life (EOL) date for a released project is something that might happen again. This killer query came about half an hour into the session, which …

  1. iron Silver badge

    Who knew Geralt of Rivia was a CentOS dev?

    1. Geoffrey W

      Hmmm...None of them look very Geralty to me. I suppose you mean the wimp at top right, but he isn't very butch, is he, and no scars. The top left guy is surely from the Beverley Hillbillies, or Duck Dynasty, or perhaps from some horror movie where hapless folk wander into some ramshackle homestead in the middle of nowhere, where machetes and chain saws await, since he does look a bit sinister.

      1. Dave559 Silver badge

        Come on, people, they're just standard unix wizards: long hair and/or beards (and mostly white and male). I'm sure few of us are ultimate god- or goddess-like beauties ourselves, and aren't we all supposed to be mature enough to accept everyone for how they look?

        Don't get me wrong, the wizard stereotype is a slightly amusing one (and, as keepers of the spellbooks, why not), but I'm not going to criticise them for it.

  2. nematoad Silver badge

    Yay!

    "...board members assured us that they were "very excited" about Stream and did not show any expressions of regret or disappointment,"

    Yeah, they would say that wouldn't they. Keep in with the bosses, after all there are jobs at stake and for that I don't blame them, but to gush all over the place how this is the best thing since sliced bread does come across as a bit forced.

    "...in effect forcing the CentOS board to go along with the plans."

    Resistance is futile.

    Now where have we heard that before?

    1. A.P. Veening Silver badge

      Re: Yay!

      Resistance is futile.

      I think it is more a case of Embrace, Extend and Extinguish, where we have arrived at the third and final stage.

  3. Anonymous Coward
    Anonymous Coward

    Open source as sales lead vs open source as competitor

    Many years ago I worked as a professional services consultant for red hat on their JBoss suite of products. There was an internal tension in Red Hat between those who saw open-source JBoss (and later wildfly) as a source of leads and sales opportunities, vs those who saw it as a competitor taking away potential paying customers. Every few years one side or the other would get an advantage and Red Hat lurched from trying to keep important features and fixes out of the opensource stream as long as possible to putting everything in opensource first. It seems that dynamic continues to this day, but with IBM at the helm they have swung further than usual. They will probably swing back the other way in a few years, once they notice the lack of CentOS users impacting their pipeline of CentOs to RHEL conversions.

    1. keithpeter Silver badge
      Windows

      Re: Open source as sales lead vs open source as competitor

      Interesting insight into corporate dynamics. I've always worked for small/medium sized public sector organisations. I see plenty of churn as a result in shifts in policy from government. I suppose this must be the commercial version, self-generated churn.

      "A beta version of AlmaLinux was released on 1 February. "The folk that are doing RHEL 8 rebuilds to keep 8 alive, we wish them the best. We love our open-source friends," said Riehecky."

      Pat Riehecky works for FermiLab (was a Scientific Linux dev) and has a significant stake in a stable rebuild OR a 'public service' RHEL licence. I'll be watching the FermiLab news page for any updates... I'm sure he was really amused at this development after the winding down of Scientific Linux and the adoption of CentOS starting with CentOS Linux 8.

      Icon: I increasingly resemble Jonny Hughes (top left on the video grab) as I stay locked down...

    2. Alan Brown Silver badge

      Re: Open source as sales lead vs open source as competitor

      Redhat has managed to continue to alienate the science community which is where most of Centos was coming from

      Their target market is things like banks, with thousands of identical almost thin-client setups and it's getting _very_ difficult to keep science softwrae running on it anymore (which is a real problem as most solar, particle and planetary physics space groups are still using it)

      I've been suggesting that our techies learn and support Ubuntu for a while but it's becoming clearer and clearer this is no longer "a good idea" but moving into "mandatory" territory (You'd be surprised how many supposed tech types through up their hands and use the "I don't understand this, it's too haaard!" card - or perhaps not)

  4. katrinab Silver badge
    Flame

    If I wanted to run beta software in production, I would install Fedora, or Windows.

    New shiny is nice, but in a production environment, I want stuff that works. So I will stick with FreeBSD, and Debian.

    1. s2bu

      You should give HardenedBSD a look at. It keeps up with FreeBSD (it isn’t a true fork), but lays down extra goodness on top!

  5. Ben Tasker Silver badge

    > This killer query came about half an hour into the session, which can be viewed here. "A question has come in about the change of the EOL for a community deliverable during a release being very unusual. Is this [a] thing that could theoretically happen in the future?"

    > ....

    > A lengthy silence ensued before Pat Riehecky from Fermi National Accelerator Laboratory, a CentOS board member since April 2020, offered: "It's hard to predict the future."

    Presumably the silence was everyone hoping one of the others was going to take that bullet.

    The answer, clearly, is - yes it absolutely could happen, just as all the ground they're "giving" on licensing to ease the transition to Stream could evapourate.

    Essentially, you need to either be thinking of changing distribution entirely, moving to one of the new community builds, or preparing for the costs of entering the RHEL licensing ecosystem.

    1. Alan Brown Silver badge

      "or preparing for the costs of entering the RHEL licensing ecosystem."

      Those costs aren't JUST the licensing. It's RHEL being being seriously crufty in a bunch of areas and the RHEL rpm environment not being something you can just drop XYZ pakaging RPM into

      EG: Just because SSH in RHEL7 says its 7.0.2 doesn't MEAN it's 7.0.2, it may incorporate chunks of later versions (including pieces of SSH 8) AND still be crippled by pieces of SSH 6 as well. (I'm bringing this one up as it's a real world example and it's major security issue. The only way to disable dangerous ciphers is to hardcode the exact list of what you'll accept, which in turn means you need to look at every update to see if things have been changed/new ciphers added

      Of course researchers being researchers, they'll see that XYZ package isn't the latest and greatest, so compile up their own version and locally install it, leading to all kinds of library clashes and the inability to keep rpms portable across systems unless extensively customised - which defeats the purpose of using a distribution-based system in the first place

      You can tell people not to install latest "everything" and build against that until you're blue in the face. It doesn't sinnk in until the $2million contract you have with ESA is being cocked up because such decisions mean the software you supplied them won't work in their standardised environments (yes, really).

  6. Down not across Silver badge

    Trust

    That question to the CentOS board about changing the EOL of a released project was really about trust, and it was one that board members chose not to answer.

    And by not answering, they have answered it.

    1. Fruit and Nutcase Silver badge

      Re: Trust

      Give it a few years and Red Hat development will move to India per IBM standing orders elsewhere in the diminishing empire.

      Trust IBM - yeah right

    2. Electronics'R'Us Silver badge
      Megaphone

      Re: Trust

      It's not just the software / OS world that is built on trust; the best vendors in technology realise this.

      I had $BIG_VENDOR decide to EOL a bunch of products with no possible last time buy. They got moved to the 'Do Not Use' category really quickly.

      In high reliability and other areas that need true long term support (think decades) the trust in the vendors to continue to support specific parts becomes a major part of the decision making process at the design stage.

      If a part (especially in some areas) becomes EOL, it may be necessary to re-qualify the equipment (and even partial requalification can be $$$$$$$).

      A vendor who has lost the trust of any of the larger companies in those areas (and others) will slowly but surely lose design wins to companies who actually keep their word. Word does indeed get around - it is not that huge an industry and many of the people in it are pretty connected even if they are in competition.

      Design win on a product with a long lifetime requirement == long term sales. Keep your word and design wins++. Rinse and repeat.

      Note that the vendors do not have a contractual arrangement to supply parts for that amount of time (unless you want to use a die banking service - $$$$) but they are acutely aware that the trust of the developers is their single biggest asset.

      Lose that trust and those designers will never use your products again wherever they happen to be working.

  7. Blackjack Silver badge

    Guys just move on to another Linux distro, Debian for example may need better wiki information but can fit any of your needs.

    1. Anonymous Coward
      Anonymous Coward

      No it can’t, not when 3rd party vendors won’t certify their products on it.

      1. Anonymous Coward
        Anonymous Coward

        And yet they still certified them under CentOS, not requiring RHEL?

      2. Anonymous Coward
        Anonymous Coward

        If you ask me, even 3rd parties need to watch out.

        Now if the 3rd parties certified non-RHEL already, then they'll certify non-RHEL again, just somewhere else like Debian. This move by IBM kills off a lot of software and trickles all the way down to the desktop. Enterprise software is glaringly obvious, but also for instance software like Davinci Resolve just lost a reliable non-commercial host.

        The code battle is ironic a bit and worriesome further. While granted not identical, consider SCO vs. IBM. Now IBM vs. CentOS? Can IBM see a future of claims against CentOS forks similar to SCO's claims? I know that sounds crazy but, so did the SCO vs. IBM and that still happened.

        1. TrevorH

          This had nothing to do with IBM. Red Hat are quite capable and willing to shoot their own feet.

      3. Smirnov

        No it can’t, not when 3rd party vendors won’t certify their products on it.

        There's still openSUSE Leap, after all SUSE is the #2 enterprise Linux vendor after RH and most ISV applications that are certified for RHEL and CentOS are also certified for SUSE (for example, SUSE Enterprise Linux is the primary platform for SAP HANA).

        If you have to get support then SEL will save you some bucks over RHEL support, and at least for us SEL support has been excellent.

        1. keithpeter Silver badge
          Windows

          Re: No it can’t, not when 3rd party vendors won’t certify their products on it.

          @Smirnov

          I know little about the SUSE world. Is it meaningful for an ISA to certify software on OpenSUSE for intended use on SEL? Or does it work the other way round (software certified on SEL will work OpenSUSE guaranteed?)

          Icon: I'm a clueless end user, just interested

        2. Alan Brown Silver badge

          Re: No it can’t, not when 3rd party vendors won’t certify their products on it.

          " after all SUSE is the #2 enterprise Linux vendor after RH "

          Suse have proven themselves quite willing to toss non-german speaking customers under a bus and run away (not even responding to quesries on the issue from Novell head office)

          We're talking serious and permanent breaches of trust. They're on my "Never do business with this company again" list

          1. handle handle

            Re: No it can’t, not when 3rd party vendors won’t certify their products on it.

            SUSE hasn't been under the Novell umbrella for a number of years now. (10+?)

      4. Anonymous Coward
        Anonymous Coward

        If enough organisations start moving to Debian, software developers will certify their software for it.

        If their RedHat-alike customer base starts dwindling away, they'll realise that they'll have to "follow the money", if they want to keep their income stream.

        (a different AC)

    2. Smirnov

      Debian for example may need better wiki information but can fit any of your needs.

      No, it can't, and that's not just because of the lack of ISV certificates. There are other issues, for example the fact that Debian is maintained by a community with a long track record of focusing more on empty activism than to improve their software quality. Part of the problem is also that many contributions come from people who don't know how to write proper code or even a bug report, and that shows. The ridiculous OpenSSL fuckup was just one of many miss-steps of Team Debian.

      Then there's the "support" cycle which is just 2 years through the Debian maintainers and an additional three years through a completely separate (and much smaller) group. Which gives you 5 years of support through volunteers. Which, again, means the quality varies a lot and fixes can take a long time to ripple through (the fact that Debian sticks with mostly very old package versions doesn't help here, too, and there's not much backporting in the way RH does).

      Debian certainly has its place, and might be a good starting point if you want to create your own distro or hosting platform from scratch, or are into fiddling in the guts of open source. Not for a business for which the OS is merely a tool to run specific applications.

      1. keithpeter Silver badge
        Windows

        Re: Debian for example may need better wiki information but can fit any of your needs.

        "The ridiculous OpenSSL fuckup was just one of many miss-steps of Team Debian."

        Anything more recent than 2006 that is specific to Debian?

        Icon: I'm a clueless end user, and I'm glad you are happy with Suse for your customers/users

  8. TrevorH

    Plus the CentOS board have no power to make any decisions about it at all so their "It's hard to predict the future" really translates as "we do what we're told".

    The majority of CentOS board members are Red Hat employees and most of them, if not all of them, are nowhere near the C-level execs needed to make such decisions. Some others are not RH employees but when the vote goes 7:2 (or whatever) it's not hard to know who voted where...

    1. MrBanana Silver badge

      They are not Red Hat employees, they are IBM. This move, and doubtless more to come, was entirely predictable.

      Offshoring will commence in 3... 2... 1...

  9. needmorehare
    Linux

    Are people sure they get what Stream is?

    I used CentOS Stream before it was "the focus" and I think people might be freaking out a bit here and misinterpreting what it is. It's patches which form a gradual set of changes between minor releases. Minor point releases are like service packs when it comes to server components. Contrary to what folks say, it's not a development system like Fedora or a rolling release system at all. If anything, it gets your real world bugs fixed quicker without breaking ABI compatibility for components which are marked as ABI-stable.

    Put another way:

    * How many people running servers would delay switching from say 7.1 to 7.2 and risk compromise?

    * How many did NOT have their yum config simply specify 7 rather than a point release anyway?

    * How many people are testing their patches in a dev environment, then staging them for production?

    If you don't do the first two but follow the best practice of testing patches anyway, then Stream is less work to maintain compared with point releases anyway because changes are smaller, rather than in one big bang, getting your real world problems fixed quicker when you report them, despite not being a paying customer.

    The only down side I can see is that you have to migrate to the next major Stream every 5 years rather than every 10. However, with the rate of change Linux undergoes, I'm not so sure that's a bad thing these days. CentOS 6 was hideously broken on newer hardware towards the end of its lifecycle and 7 is already getting that way.

    1. TrevorH

      Re: Are people sure they get what Stream is?

      The downside that you are missing is that with the old style CentOS, you could plan for a point release coming along and including new things that would break your system. So you knew when to watch out for breaking changes and could plan for them. In the new scheme of things you will now get breaking changes whenever Red Hat feel like pushing them.

    2. Anonymous Coward Silver badge
      Boffin

      Re: Are people sure they get what Stream is?

      It's not about what Stream is; it's about what they're positing it to become.

      They're most definitely positing it to become the staging/development area for RHEL. That's an uncomfortable place to be in a production environment.

      1. FrankAlphaXII

        Re: Are people sure they get what Stream is?

        Its not just uncomfortable, its a downright insane place for a production environment. No one who values their clients or their job is going to use it in production because its only a matter of time that some untested piece of beta software breaks something very important.

        Fundamentally, CentOS stream is a beta where Fedora is an Alpha. It is never a good idea to use beta software in production anywhere outside of a UAT environment with power users, QA testers and developers that know they're testing something and that it might break. That's different from an unsuspecting client who expects the software they're using in production is stable and probably won't unexpectedly break just because the vendor pushed out something that broke a mission critical piece of software.

    3. williamsth

      Re: Are people sure they get what Stream is?

      What a silly comment.

      Especially:

      "How many people running servers would delay switching from say 7.1 to 7.2 and risk compromise?"

      If there's a known security vulnerability, it gets patched and backported pretty quickly. If there's a bug, those also get backported. The idea is your software version stays the same and you can PLAN for the bigger updates. You obviously have no idea how to run servers in a production environment.

      1. needmorehare
        Facepalm

        Re: Are people sure they get what Stream is?

        Non-current minor releases don’t get security updates backported because CentOS doesn’t include EUS errata. Without EUS, holding off on a minor point release means no more errata whatsoever. That’s why people have their repos set to 7 and not fixed to a given point release, so they get the latest patches when they are available, precisely because patches are NOT backported to non-current point releases on CentOS in the default Base repositories.

        So let me reiterate: How many people running servers would delay switching from say 7.1 to 7.2 and risk compromise?

    4. Anonymous Coward
      Alien

      Re: Are people sure they get what Stream is?

      The answer to this is that when some vast half-working system is finally persuaded to actually work, when arranging downtime takes hundreds or thousands of hours of people's time and when outages cost them business or, worse, end up on The Register (or, worse, the TV) people become very, very frightened about deploying upgrades indeed. And when the existing system is deployed on x.y then you very definitely want and new OS instances to also be at x.y, not x.z.

      Yes, many of these things are bad practices, but they are also what life is actually like for many organisations. The flip side, of course, is that these people are already on RHEL, not CentOS. The people who were doing this on CentOS were probably alway going to get screwed, and they just have.

  10. Sanguma

    Problem

    If RedHat's already got a Fedora Server, then introduces a CentOS Stream to fill exactly the same niche, won't they run the risk of confusing their customers? Confusing customers - what a way to go - they usually go elsewhere. Anyway, they've sold out one highly valuable brand and I doubt they'll be getting its equivalent back any time soon.

    On the upside, AlmaLinux beta's usable. I haven't tested Rocky Linux but it still hasn't graced the download links yet.

    1. FrankAlphaXII

      Re: Problem

      They're not saying this, but the way I'm looking at it is that Fedora Server fills that traditional role that an alpha did back when the software lifecycle made sense, and that CentOS Stream fills the beta role. They're both nothing anyone with sense is ever going to use in production.

      IBMhat thinks that they're going to drive people purchasing RHEL by doing this, but its going to alienate people who were using CentOS as a relatively stable environment to try to push people to buy RHEL by showing how well it worked

  11. Anonymous Coward
    Anonymous Coward

    CentOS 8.0 Linux, built from the same sources as Red Hat Enterprise Linux (RHEL) 8, was released on 24 September 2019 with a projected EOL of May 2029.

    In December 2020, community manager Richard Bowen posted about the "shifting focus" to CentOS Stream, announcing that "CentOS Linux 8, as a rebuild of RHEL 8, will end at the end of 2021."

    Oh wow, somehow I missed that they were doing this.

    Red Hat just got themselves added to my "company cannot be trusted, do not use" list, right next to Microsoft.

    1. MrBanana Silver badge

      'Red Hat just got themselves added to my "company cannot be trusted, do not use" list'

      They made that list the instant they got bought by IBM.

      1. Alan Brown Silver badge

        the dead hand of IBM has been obvious for a while

        1. EnviableOne Silver badge

          the red has been going purple for a while

          this has Blue fingerprints all over it

  12. Adair Silver badge

    Are people paying for this?

    If they aren't, and there's no contract been signed, they really need to just suck up what they get - after all, that's what they're not paying for.

    1. Anonymous Coward
      Anonymous Coward

      Re: Are people paying for this?

      If I go back on my word, then I'm a prick who should not be trusted. Regardless of whether we have signed a contract or not.

      All signing a contract does to change the situation is that it gives you some recourse if I do turn out to be untrustworthy.

      But the lack of a contract doesn't mean that ethics somehow don't apply.

      So I'm not sure what point you're trying to make here? That the people who have been misled have no recourse? Or that they were naive for expecting others to act in good faith? Or perhaps you're saying that there's no problem with this massive breach of trust because the law doesn't care about ethical behaviour, so to hell with ethics as a concept and tough shit to everybody who has been misled?

      1. Adair Silver badge

        Re: Are people paying for this?

        The point I'm making is that people who opt to run mission critical services off the back of a service that is provided for 'free' are not in a strong position to complain if the ground shifts under their feet.

        'Promises' are easily made and easily broken, for all sorts of reasons - good, bad and through force majeur.

        If I'm relying on someone else's goodwill for a service I had better make jolly sure I have a 'Plan B' in place, and am willing to 'pay the price' of free-loading on someone else's goodwill, because in this life things can suddenly change. Whining never impresses.

        1. Anonymous Coward
          Anonymous Coward

          Re: Are people paying for this?

          All true. However, the other poster's point is also valid.

          Adopting any open source (especially free-as-in-beer) software requires trust that the provider of that software won't suddenly turn around and shaft you. Cutting off early adopters of CentOS 8 the way they did means Red Hat are no longer worthy of that sort of trust. Red Hat may have acted entirely within their legal rights, but they've hosed their reputation (and possibly lost a source of sales leads) by doing so.

          1. Adair Silver badge

            Re: Are people paying for this?

            I'm certainly not condoning what's happened. I'd be pretty fed up myself, but people in that situation have taken a known risk. They are now acting surprised and seemvto have no contingency plans in place.

            1. Anonymous Coward
              Anonymous Coward

              Re: Are people paying for this?

              Yeah, technically you're right and your point is valid. But I don't think it's unreasonable for people to be surprised or annoyed in this instance. Most people using it do have an (unwritten) contingency plan for this scenario: switch to another distro. But I see nothing at all wrong with grumbling that you were lied to and so you now have to enact that plan - a bunch of work you weren't planning on doing.

              We all know that that planet-killer asteroid is on it's way, but I bet you'll still be surprised if it lands on your house tonight.

              And I'm also willing to bet that when it does land on your house, you're not going to think me saying "well you knew mars was right there the whole time and you knew how the dinosaurs died" is very helpful...

              1. Adair Silver badge

                Re: Are people paying for this?

                Some risks are worth planning for, others you just have to say 'Que sera, sera'.

                The killer asteroid strike - shit happens.

                The distro maintainers doing a U-turn on a 'promise' they made to... actually to 'no one' because everyone using the distro is doing so of their own choice and at their own risk - I have to assume it's probably going to happen within my time of use, and if not that then some other distro-screwing tragedy. I can wail at the moon when that happens (but no one gives a shit), AND have my Plan B in place.

                1. Anonymous Coward
                  Anonymous Coward

                  Re: Are people paying for this?

                  When that asteriod strike happens I'm gonna be right here unhelpfully telling you that you should have had a plan in place :P

                2. jtaylor Bronze badge

                  Re: Are people paying for this?

                  The distro maintainers doing a U-turn on a 'promise' they made to... actually to 'no one' because everyone using the distro is doing so of their own choice and at their own risk

                  Redhat did make a promise: they promised to provide updates for 10 years from initial release. They made the promise for both CentOS and RHEL, and reneged.

                  I think your point is that people and businesses that use only CentOS saved on license and support costs by assuming the risk themselves. If they no longer like that risk, they can start paying for support or find someone else to provide a similar service for free. That seems reasonable.

                  Some people run CentOS to develop for a target of RHEL. They are not paying Redhat, but their customers/employers are. Some businesses run CentOS where they will self-support, and RHEL where they want outside support. In both of those situations, CentOS is part of the overall value proposition that RHEL offers and for which they are paid.

            2. hoola Silver badge

              Re: Are people paying for this?

              Taking a known risk is true however one looks at the history around CentOS and as far as I can tell there is no indication that this was going to happen.

              It certainly has not been at the top of our list of risks when we rolled out thousands of CentOS nodes in place of Scientific Linux.

      2. Anonymous Coward
        Terminator

        Re: Are people paying for this?

        I think if you're dealing with commercial organisations and you assume that they are not pricks who should not be trusted you are being touchingly naïve.

        1. Anonymous Coward
          Anonymous Coward

          Re: Are people paying for this?

          Maybe, but I think you should give people a chance to demonstrate their trustworthiness. Or lack thereof.

  13. sabroni Silver badge

    They've shown how much repect they have for non-paying customers

    You now need to assess the risk they pose to your business and react accordingly.

    If your plan runs for 10 years and you don't like paying for a supported OS to host it I'm not sure what your options are. I'd be questioning the business plan if it was me.

    1. Anonymous Coward
      Anonymous Coward

      Re: They've shown how much repect they have for non-paying customers

      If you are not paying you're not technically a customer. "Users of CentOS" is a better description than "non-paying customers".

      1. Anonymous Coward
        Anonymous Coward

        Re: They've shown how much repect they have for non-paying customers

        "Potential sales leads" is another apposite term. Presumably RH didn't think those leads were valuable enough to continue with the status quo.

  14. TVU

    "Q?: How can we trust you now?"

    A!:You can't because that horse has already bolted. What you can do is use and support one of the community replacements for CentOS such as Rocky Linux.

    The moral of this tale is that big tech companies ought to think first before they act and ask questions of themselves, "How will this affect trust in our brand if we do this?".

  15. Binraider Bronze badge

    Jump ship already. IBM have made their intentions clear; harassing the CentOS board isn't going to change IBM from being IBM.

  16. jason_derp Bronze badge

    So, it has to be good

    "The board and Red Hat want Stream to succeed. If we make it terrible, it will not."

    Mmhmm. That's certainly true of every product squeezed out of a corporate orifice. No bad leakages have ever managed to stain the drawers of anyone unfortunate enough to get too near. It's all good and logical. Yes.

  17. FuzzyTheBear
    Unhappy

    Trust ? o.0 ?

    IBM bought RedHat and RedHat became instantly untrustable in any way.

    redhat being bought meant the end of an era.

    now we see how much that was true.

    time to ditch them as they ditched us.

    See , once trust is broken .. there's no way back.

  18. xosevp

    Is this thing that could theoretically happen in the future, from your perspective?

    Exact time of the "question", 34:00: https://youtu.be/CXFrBv1wwvg?t=2040

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021