back to article How do you fix a problem like open-source security? Google has an idea, though constraints may not go down well

Google has proposed a framework for discussing and addressing open-source security based on factors like verified identity, code review, and trusted builds, but its approach may be at odds with open-source culture. The security of open-source software is critical because of its wide adoption, from the Linux kernel on which …

  1. Paul Crawford Silver badge

    Hmm, so they want OSS projects to behave like paid projects, but without actually paying them?

    Goals may be noble, but really having some cash thrown towards critical projects would be a good thing as well. Before they cause a problem and emergency work starts like openssl...

    1. pavel.petrman

      Especially from the likes of Google, who very much like to take and are much more hesitant to give. See Chromium vs Chrome or the OSS Android Base. All in all as bare shells as legally possible to contain all the license requirements and stop them from actually working as intended, and first after you've done that, slap all the sweet sweet money making icing on top of that.

      1. cdegroot

        Or see the apparent fact that they have vetted, hardened forks of important packages that they keep to themselves. Google is only interested in Google and this is an excellent example.

    2. Anonymous Coward
      Anonymous Coward

      " they want OSS projects to behave like paid projects Google's corporate assets"

      Virtual World: "Ummm, ever heard of Google?"

      Real World: "Sure, sure... I've heard of Trojan Horses."

  2. Gene Cash Silver badge

    Lots of questions

    * What is a "vulnerability database" and why do we need multiple ones? What's the difference between this and the list of vulnerabilities that CERT maintains?

    * "A notification system for the actual discovery of vulnerabilities." - WTF? Isn't that CERT?

    * "That no changes are made to critical open-source software" - who determines "critical software"?

    * "an attested build system" means "a Google build system" right?

    1. Paul Crawford Silver badge

      Re: Lots of questions

      "an attested build system" means "a Google build system" right?

      If you ever had the sad misfortune to be unable to avoid using GNU Radio you would be pleased if the code would actually build out of the repository. A project that needs to invent its own build management tools is a sh*t-show in the making,

      1. doublelayer Silver badge

        Re: Lots of questions

        Excellent point, which is why complex build systems can be a problem. One that's there for convenience, automating the process of test, package, sign, and release is fine. One where the build system is intrinsically connected to the build such that you can't easily build without it is bad. Unfortunately, I have seen build systems like that more often than I'd like.

        1. Pascal Monett Silver badge

          If it's convenient, then it's not secure.

          That's the problem with security : it's necessary, but it's a bloody nuisance.

          1. doublelayer Silver badge

            It's not true. Some security is at odds with convenience, E.G. passwords or keys or multi-factor authentication. Some security though doesn't have to conflict, and if it's to be used, it must not. Security checks can be integrated well with existing processes and systems, or they can be limited to one that's new and inconvenient. Either way, they'll check for security problems equally well. If they're easily available already, they'll get used more often. In this case, the security outcome of running the stuff is unrelated to convenience of running it, and overall security is directly proportional to convenience because convenience is directly proportional to usage. The only thing that's more difficult there is the original integration with existing processes, which will take longer, but it's often the price that must be paid.

      2. Greybearded old scrote Silver badge

        Re: Lots of questions

        Have you read the recent reports about how difficult it is to build Chromium? Let me see, who runs that one?

    2. Michael Wojcik Silver badge

      Re: Lots of questions

      * What is a "vulnerability database" and why do we need multiple ones? What's the difference between this and the list of vulnerabilities that CERT maintains?

      CERT/CC maintains the CVE registry and database, and CVEs themselves are generally pretty short on technical information. The actual vulnerability database, in the US, is the NVD, maintained by NIST. Other organizations maintain their own mirrors, often with additional information.

      I think most IT security professionals would agree that the CERT/CC CVE registry is not sufficient as a vulnerability database. It's useful, but it's far from the only one that I use, and the same is true of all the other IT security people I know.

      * "A notification system for the actual discovery of vulnerabilities." - WTF? Isn't that CERT?

      Vulnerabilities are often published well before they have a CVE assigned, or at least before it's updated with details. Many of the vulnerabilities in the NVD go for months or years without actionable information.

      And because of the requirements of the CVE program (e.g. needing a CNA), some open-source projects don't use CVEs.

      * "That no changes are made to critical open-source software" - who determines "critical software"?

      Yeah, I don't see how this one is going to fly, for a number of reasons. Also, while I don't want to subscribe to tu quoque, the Google pot is talking some smack about the open-source kettle here.

      * "an attested build system" means "a Google build system" right?

      I haven't read the blog post, but no, not necessarily. There are some pretty clever proposals out there for attested build systems, such as CHAINIAC.

      CHAINIAC isn't something a project could just drop into its existing build process; it's complicated even in theory (though if you're comfortable with things like general Nagle DAGs, not particularly complicated) and there are scalability problems, particularly around the code reviews (though I think that's solvable). And, of course, it's not perfect; there's no such thing as perfect security. But it's an example of how attested builds could be done, and how they'd prune a lot of the software-supply-chain attack tree.

      Should this be at the top of the priority list for software-security issues we need to address? That's a separate question. Supply chain security is a big problem, but there are a lot of big problems in software security. Hell, if we could just get more of the prominent open-source projects to regularly run static and dynamic analysis I'd be a lot happier.

  3. Adair Silver badge

    Inevitably ...

    some people are starting to think that they 'own' GNU/Linux, because ownership is what they are all about.

    The day may indeed come when 'Linux', as a software stack, effectively becomes a corporate product, but the thing they cannot own is the philosophy that drives 'FLOSS'. If the corporate ossification and monetisation of 'Linux' does ensue the spirit driving 'FLOSS' will simply move elsewhere.

    There will always be a place for a wild IT frontier where people with an itch to scratch and a generosity of spirit to share their solution will set it free to be picked up by whoever wants to take it and develop it to scratch their particular itch.

    1. Paul Crawford Silver badge

      Re: Inevitably ...

      Thing is, we already have "corporate Linux" in the form of distros by Red hat, Canonical, etc. They are the ones who should be doing the code review and build verification before it becomes "mainstream" to paying customers.

      Customers who expect that without paying have an interesting prespective on the world.

      1. Adair Silver badge

        Re: Inevitably ...

        The problem is the friction between those who think GNU/Linux is 'just a free version of Windows' and those who think that 'GNU/Linux' is a manifestation of a particular philosophy of 'freedom'.

        One of the 'costs' of that freedom is that people walk away from stuff they aren't interested in - even if it really matters to others or themselves. Tough on everyone, but that's the reality. Sure someone can pay to have the work done, but if they aren't prepared to share why are they even in the room? There are perfectly good proprietary systems available just down the corridor where 'ownership' is the name of the game, and they can screw over everyone else as much as they like.

        1. Pascal Monett Silver badge

          I do not recall that it is forbidden to do proprietary software on Linux, just as it is perfectly possible to do FOSS on Windows.

          Of course, if you do go proprietary on Linux, then you have to go all the way and build your own library stack, because woes to the guy who uses open libraries to make his proprietary software.

          1. Adair Silver badge

            Of course it's not forbidden - people can do what they like, within the limits of the law.

            But, I don't think that's the point. As I tried, probably poorly, to point out GNU/Linux provides a field of play for a particular philosophical approach to 'doing software' to be expressed. Not that it can't be expressed elsewhere - and it certainly is - but currently the 'field of play' offered by GNU/Linux is the most expansive and receptive to allowing that philosophy, and its practical outworkings, to be expressed.

            Even so, given that this is a human endeavour, being conducted by a whole gamut of people in all their glorious eccentricty and contrariness, not to mention selfishness and greed, the whole thing is unsurprisingly hairy, uncontrolled, and highly inconvenient at times.

          2. Michael Wojcik Silver badge

            Of course, if you do go proprietary on Linux, then you have to go all the way and build your own library stack, because woes to the guy who uses open libraries to make his proprietary software.

            No, you don't, as evidenced by the success of proprietary software that runs on Linux.

            There's the LGPL. More importantly, there is a lot of proprietary, closed-source software running on Linux. You might claim that some or all of it is in violation of some license; but until that's established in a court, that's irrelevant. For it to be established in court, someone with standing has to sue, and it doesn't appear that anyone's inclined to do that.

          3. thejynxed

            Or just use a BSD or MIT license and avoid the cancer known as GPL.

      2. Anonymous Coward
        Anonymous Coward

        Re: Inevitably ...


        Quote: "...Red hat, Canonical, etc. ... are the ones who should be doing the code review and build verification ... Customers who expect that without paying ..."

        Well....I was a paying customer for Red Hat from RH5.2 till RH9...and happy to pay for excellent software. Then Red Hat abandoned their retail editions and started Fedora. At this point I had two choices - either pay for RHEL, or use Fedora "without paying". Since an RHEL subscription was too rich for me....I chose Fedora.

        The Fedora deal is a reasonable bargain:

        1. No support - I am responsible for the (four) machines which I operate.

        2. Red Hat get feedback from me and people like me so that Red Hat can improve the quality of future releases of RHEL.

        But to characterise my choice as freeloading is a cheap shot. Not least because I would still pay the approximate $40 price for each (now unavailable) retail release of Red Hat.

    2. nematoad Silver badge

      Re: Inevitably ...

      Aye, look at the titles of those proposing this:

      VP of infrastructure

      distinguished engineer

      principal software engineer

      program manager

      product manager

      Looks like an empire in the making complete with bureaucratic tendencies.

      The suits are moving in and will take a lot of shifting.

      Still this is FLOSS we are talking about and they may have bitten off more than they can chew

      1. Anonymous Coward
        Anonymous Coward

        Re: Inevitably ...

        I think you've almost nailed the issue. Only it isn't the job titles, its the need to add more people/processes.

        The problem for these projects is that they are effectively abandonware BUT still key dependencies for other projects. Adding in "requirements for a new process" doesn't address the issue of insufficient people maintaining the code, at best it just highlights the issues sooner.

        What they need is a role created for "Principle Dead horse Flogger" to make the dead horse work harder and make less mistakes. I would include training on security issues but with all the bugfixes and other changes we have lined up, that dead horse won't have any time for training.

        Or the large companies that are so worried about these issues could try helping the dead horse with additional resources or just burying it.

        1. Steve Davies 3 Silver badge

          Re: abandonware

          Google is very well known for 'abandonware'.

          Any takers for how long this [cough][cough] initiative will last before it is quietly shelved?

          IMHO, this is just another of Google's many attempts to control the IT world or at least major parts of it.

          If they (google that is) can monetize this then they will and they'll do it big time.

          1. John Sturdy

            Not everything they do will centre on monetization

            I don't get the impression this is about getting them more income... more like worrying about their existing income streams getting broken by a massive production failure, caused by a change to something they import, and leading to paying customers moving away from them.

      2. Michael Wojcik Silver badge

        Re: Inevitably ...

        What a bunch of ad hominem bullshit. Try making a real argument.

        1. jake Silver badge

          Re: Inevitably ...

          Observations based on past actions leading to conjecture about the future doth not make ad hominem.

        2. Anonymous Coward
          Anonymous Coward

          Re: Inevitably ...

          "What a bunch of ad hominem bullshit. Try making a real argument."

          Shall I try small, simple words then?

          Too few people.

          These projects are dying because of a lack of developers. Adding processes to address security adds work without addressing the resource issue so is doomed. Anyone suggesting this as the answer doesn't understand the problem.

  4. doublelayer Silver badge

    You want all this, Google

    If you want all of these things, how about you do them. And no, that does not mean you give free Google Cloud credits to the projects. It means you construct any standard protocols you want and work to get them adopted. And when people don't adopt them because they're arcane and ill-supported by anyone but you, you change them so they do what you and others want. And you support and secure any new databases or systems you think people need. If they're so great, you can expect that others will adopt them voluntarily and continue to support and advance them. If you construct them to lock in developers, expect to be snubbed.

    Free software developers work because they see a need and they're generous people. They don't work to keep your salary coming in, and they're not going to change everything because you don't like what they're doing now. Don't try to force anything on them unless you want them to hate you. You want to improve things, do the work.

    1. Doctor Syntax Silver badge

      Re: You want all this, Google

      The downside of that is that this is Google. The Google that simply dumps any plaything it gets tired of.

      1. vtcodger Silver badge

        Re: You want all this, Google

        The Google that simply dumps any plaything it gets tired of.

        I don't think Google is likely to tire of dealing with security. They may eventually give up because the problem is too intractable. But security of the common computing/communications structure is pretty central to Google's business model. No security on the internet means not much cloud and eventually severely restricted digital advertising and a reduced bottom line. If there is anything Google cares about, it is its bottom line.

  5. jake Silver badge

    Turn about's fair play.

    "Google, it turns out, does not entirely trust the usual open-source repositories and package managers."

    On the other hand, show me a FOSS aficionado who trusts go-ogle-abet with ... well, with anything at all.

    "Try telling the leaders of various projects like libpng, libjpeg-turbo, openssl, ffmpeg etc that they are not allowed to make 'unilateral' changes to their own projects just because they are critical software in the FOSS world,"

    Worse, try telling them that they are not allowed to make changes to their own FOSS project because Alphabet is worried about go ogle's bottom line.

    I'm looking forward to an open source license that states something along the lines of "Free for anyone, anywhere, to use for anything ... except by Alphabet, its subsidiaries (e.g. go ogle, et alia), it's officers and employees and anything that comes into financial contact with them in perpetuity". Dunno how it would hold up in court, but it would make me giggle ... and might, just might, get the point across.

  6. Howard Sway Silver badge

    factors like verified identity, code review, and trusted builds

    You don't need to verify the identity of the person who did the change when you can look at exactly what was changed. Especially when you're a company that knows so much about everyone anyway, and would probably start flagging up contributions if that person failed one of your secret checks linked to your own database for any reason, however minor.

    If you don't trust the code, you're free to review it (and if the project has many contributors, you know lots of eyes already have).

    If you don't trust the build, build it yourself.

    Problem solved. Exactly how it was intended to be.

    1. bombastic bob Silver badge

      Re: factors like verified identity, code review, and trusted builds

      You don't need to verify the identity of the person who did the change when you can look at exactly what was changed.

      Correct. And it's theoretically possible to grab a source tree based upon a specific commit, or a specific version, if you need to [for patches, at least].

      What you do *NOT* need is "da bleeding edge newest feature-creep-laden" version, EVERY! SINGLE! TIME! you update.

      What I see as the problem: Someone "not you" is deciding what version of code is being used by your application or operating system.

      What I see as the fix: enable software authors and package maintainers to depend on installing older (or newer) versions of critical libraries if they want to, with forked security patches, as needed.

      What needs to happen to make this work: either static link the binaries and manage them individually, or else use something similar to a "container" or "run environment" to be installed with specific package versions assigned to critical (or otherwise incompatible) packages. Ubuntu may already have "a mechanism" for this....

      Just because a shared lib exists on the OS does not necessarily mean you MUST depend on it. Your critical package dependency tree could easily depend on versions that have been PROPERLY VETTED, and maintain security patches on.

      To some extent, an LTS release will do this. By fixing code versions in stone, and ONLY patching vulnerabilities, you generally are NOT introducing NEW ones. Prior to a Linux release that's LTS, responsible package maintainers would vet all of these things (and prevent breakage).

      Long ago I realized that you can NOT rely on shared libraries being updated to NOT break YOUR application, especially if you ship binaries. To some extent Linux handles this by versioning the names of the shared libs, but this does NOT correctly patch security vulnerabilities if you update ONLY THE NEWER VERSION of that shared lib. This patchwork "must be a shared lib" approach is flawed in this way. As I see it, for any critical application, you either include your own libs in the build process and link them statically (or dynamically with unique-to-your-application names), or you use specific LTS versions of the libraries as dependencies for your shipped binaries, thus ensuring that security patches (and not "FEATURE CREEP") are the only things done to them.

      Things broke HORRIBLY WRONG when everyone "suddenly decided" it was necessary to maintain bleeding edge "feature creep laden" versions of EVERYTHING, and to CONSTANTLY HAVE A MOVING TARGET, in lieu of "stable, well tested, rarely changes except to fix serious problems" .

      Change and re-inventing the world, because, "new, shiny" - SO HIGHLY OVERRATED!!!

      1. jake Silver badge

        Re: factors like verified identity, code review, and trusted builds

        "when everyone "suddenly decided" it was necessary to maintain bleeding edge "feature creep laden" versions of EVERYTHING"

        Not everyone. See slackware-stable ... 14.2 was released on June 30th, 2016. Pretty much nothing but security patches and the odd prudent upgrade and addition ever since. Solid as a rock, too, it just keeps on ticking, quietly going about it's business and staying out of my way. Can't ask for much more than that in an OS.

        You bleeding-edge folks can still get your fix with slackware-current. Rumo(u)r has it that 15.0 is about to go into Beta. FWIW, I find -current to be just as stable as -stable, but I wouldn't trust it for anything important.

    2. Michael Wojcik Silver badge

      Re: factors like verified identity, code review, and trusted builds

      Right, which is why Seggelmann's mistake in his implementation of DTLS Heartbeat in OpenSSL was caught immediately.

      Oh, wait.

      Many eyes is a fine idea. In practice it hasn't done a damn thing for software security.

      But maybe if we all just believe harder....

  7. rcxb Silver badge

    Huge hassle to address non-problems

    This sounds like "security theatre." Google wants to pin down the identities of all code contributors. What percentage of exploitable bugs in open source code would you say were introduced by malicious actors thus far? They've put together an onerous system that's going to drag down everyone, just to make sure that 1 in 100,000,000 thing doesn't happen.

    That would be a great idea to implement at SolarWinds, but just rubbish for open source projects.

    1. yetanotheraoc Silver badge

      Re: Huge hassle to address non-problems

      "Google wants to pin down the identities of all code contributors."

      That's google's hammer, looking for a nail.

  8. Duncan Macdonald

    Who knows ?

    Many programs import packages that import packages that import packages ...

    Just trying to get a full listing of the sources that go into a particular build can be difficult - and can change the next day when a package 5 levels down includes a different even lower package.

    When the lower level packages are imported at runtime (eg DLLs or Javascript modules), a bug free program may turn into a wreck without the program source being altered due to a change in the imported package (often with minimal documentation of the change).

    I do not know a good solution - using a private repo like Google and checking alterations before they are accepted into the private repo is possible for a company as large as Google but is impractical for small developers. However the current system makes it too easy for insufficiently tested changes in one project to crash another project in a different company.

    Possibly Google could afford a few hundred millions a year to check all the sources in GitHub etc for errors!!

  9. Pirate Dave Silver badge

    It's almost

    like Google is building up to when they say "Why sure, we'll take on this all-important, but monumental, task of making sure no FOSS libraries are updated without 'approval' of the 'community'. We love to do our part. Don't be evil. Wash behind your ears."

    Or maybe I just don't trust Google anymore...

    Or maybe I really just don't trust ANY techno companies anymore...

    Old. Cynical. Yep.

    1. FlamingDeath Silver badge

      Re: It's almost

      Age and cynicism aren’t even required these days

      You just gotta be awake to see the shitshow

      I dont trust any of these tech companies

      You know the world is fucked when the likes of James

      murdoch is on the Tesla board

      And Gates, looking evil as fuck, and never too far away from a syringe

  10. Pete 2 Silver badge

    Fixing the unfixable

    > the norms of open-source culture

    and that's where it all falls apart.

    Leaving aside the OSS that is given free to the world by large, professional, companies, all the other coders write stuff because it is fun. That, and the bragging rights, are the emotional reward coders get for tossing a bunch of software over the wall into user-land.

    As for all the boring but necessary stuff that the real world needs, such as documentation, testing and support - well, that's not fun. Neither is having to formalise change control or employ standards.

    Many years ago I reported a bug in one of the FOSS office suites. The response I got back from the coordinator of the project summarised the situation along the lines that all the code was written by volunteers. They would only write software that interested them. As such there was little prospect of getting bugs fixed as none of the team were motivated to do such work.

    I do not believe there is any way to fix that problem, nor is there any leverage to implement the sort of quality control that Google is proposing.

    1. Paul Crawford Silver badge

      Re: Fixing the unfixable

      Oh there is a fix, big companies that have a vested interest could pay others to do the dull work.

      While some of the originators of the project may not care about documentation (on the ground they know it and don't care to pass it on) and they might not worry about bugs or security holes that do not impact themselves, generally if someone else is willing to do the donkey work to clean things up and make it all nice and easy to understand/build/test/review/etc, they will be happy.

      And if not, you can always tell them to fork off..and do it.

    2. bombastic bob Silver badge

      Re: Fixing the unfixable

      reporting a bug is good, but supplying a PATCH along with the bug is even better.

      1. Claptrap314 Silver badge

        Re: Fixing the unfixable

        Any intelligent user can submit a useful bug report. Only a developer with significant experience with the code in question can submit a useful patch.

        1. jake Silver badge

          Re: Fixing the unfixable

          "Only a developer with significant experience with the code in question can submit a useful patch."

          Not necessarily true. I've submitted numerous useful patches with little to no experience with the code in question. When you know the syntax of a given language well enough, some bugs stand out as if they were highlighted.

          1. Pete 2 Silver badge

            Re: Fixing the unfixable

            > I've submitted numerous useful patches with little to no experience with the code in question

            Which tells us a lot about the quality of the pre-existing code! And by extension the original coder.

            But the same applies. Writing code is the easiest part of creating software. All the drudge work is at least as valuable to the end user. But the coders do not write their applications for end users benefits.

            1. jake Silver badge

              Re: Fixing the unfixable

              "Which tells us a lot about the quality of the pre-existing code! And by extension the original coder."

              Of course. Doesn't alter my point.

          2. Claptrap314 Silver badge

            Re: Fixing the unfixable

            You were worthy

    3. doublelayer Silver badge

      Re: Fixing the unfixable

      When you don't pay the developers, you have a problem because they develop what and when they want to. They may eventually fix your bug, but if they think it's unimportant, they'll probably put it far down. The good news is that you have the source, and you can change it, and you can send it back. If the developers working for free aren't doing what you want, you can use money to fix the problem by finding someone willing to fix the problem for payment. That's not always an option for individuals, which is why some individuals will choose not to use some piece of software on the basis that it's not stable enough.

      For companies, it's quite an easy option to implement if they want things changed. Or they could also abandon the project because it's not stable enough and create an alternative, either a competing open source version to get some of the original developers to work on it or a proprietary version for the income stream. They can easily do these things, so I find it annoying when they complain about external developers without doing anything to solve the problems they have.

  11. IGotOut Silver badge

    Dear Google.

    How about making your binary blob Open Source, so people can choose the bits they want, thus avoiding having to install unwanted software that gives a wider attack vector, y'know security and all that.

    No? Thought not.

  12. Anonymous Coward
    Anonymous Coward

    Something new needed?

    So if you can't trust closed source because you can't verify the code, and/or don't trust the paid devs, and you can't trust open source in its current format because there is too much code to verify, the dependencies are too complex and too ever-changing, and you have no idea if the petite modifying the code are friend or foe, are working to their own hidden agenda etc. - What do you do?

    If we can't trust hardware manufactured in "current evil country of the moment" because the supply chain isn't secure, why is it acceptable to buy hardware from "non-evil country" but then run software on that where we can't (realistically) verify who or what has been tinkering with it?

    As others have repeatedly pointed out, no software is secure or free of bugs or vulnerabilities, and just because it's open source doesn't mean that an army of volunteers is pouring over every line of the existing code or all modifications looking for functionality, performance or security issues.

    We (in the west) are currently told Chinese-made hardware is not to be trusted. Chinese and Russian software is not to be trusted. So it seems like the current secure solution is to use western hardware and western closed or open source commercial software (where the latter does not use any code that has not been source-verified, i.e. not your regular Linux repo), and trust that the western governments are keeping an eye on people being employed in critical positions in those companies.

    So what actually is a workable solution to this, to keep the enthusiast programmers involved? Seems like Google's proposal isn't so daft? Or they could just fork everything they don't trust and employ loads of people to update that stuff for them. But their (and many other companies') business model works because of the free code/binaries they're able to use.

  13. sreynolds

    Or they could just...

    Why don't they just fork off, eh?

  14. Pascal Monett Silver badge

    Leave the FOSS developers alone, Google.

    You say yourself that you have a private repo for all the stuff you link to.

    You give up the true solution, and ignore it because you want something that is easier for yourself.

    I think the real solution is that companies that use FOSS libraries should all have their private repos, and control what it is they put into their production servers.

    If SolarWinds123 had done that, they would have spotted that there was a change in a module, checked the change, analyzed what was going on, and pulled the whistle on the issue.

    I see no need to try and impose an administrative structure with external checks and balances on people who do their work for free.

    I see a great need to impose private repo management on all those paid developers who just link to a library on Git and think that their job is done.

  15. PassiveSmoking

    Fair enough, seurity-critical components like libgcrypt need more scrutiny and oversight than the average code.

    But would the proposed system be guaranteed to have caught the recent libgcrypt memory vuln? I suspect not. So long as software is written by humans, bugs will slip past all scrutiny.

    1. jake Silver badge


      Complex code has bugs. Getting around this is difficult. About the best we can sanely do is keep vigilant and fix mistakes as we find them. FOSS seems to be doing an admiral job of this, over all, out here in the real world.

      Cue folks insisting on counting whatever it is that's dancing on the head of a HDD ,,,

  16. BinkyTheMagicPaperclip

    'attested build system'

    whilst I'm sure a build system is a possible target for attack, it's probably a less likely vector. Changing build system is usually a tremendous amount of work, and can have implications on cross platform building, etc.

    If Google thought that, e.g. the NetBSD or OpenBSD build processes were inadequate it's their action to update it whilst preserving pre-existing functionality (such as successfully building on an ancient relatively low memory system), and then see if the community adopts it.

    1. thejynxed

      Re: 'attested build system'

      It's only going to become more likely since the SolarWinds attack showed the world that major security companies can be vulnerable in their build environments and not just careless Iranian nuclear scientists.

  17. NullNix

    That's not what they meant when they said "unilateral", though, is it? The same section that talks about unilaterality also talks about all changes needing review, which *is* a good thing and isn't done remotely enough. Many of the projects cited are exceptions to this rule and do have someone not the author review every change: ffmpeg, for example.

    Google aren't saying "these projects can't change without checking with us". They're saying "these projects are security-critical so more than one pair of eyes should look at all changes". Which is, well, good. Also a bit pie in the sky, I fear... :/

    1. doublelayer Silver badge

      There's nothing wrong with asking for it, but it reads to me as if they're demanding it without wanting to help. They don't have to preach all this necessary change if they instead found some people to do the code reviews. If they sent a message like this to repo maintainers, it could work:

      Hello maintainer team, we're using your code in our products and therefore have an interest in continued security in your codebase. Therefore, we've employed some people to do code reviews on any PR to main. We'll flag security things as well as bugs that seem clear to us, following your documentation on style and requirements. It would help us if you could hold back completing PRs for a couple days while our team reviews. Of course, you don't have any obligation to do that, but we think it will result in better code for both of us. Thanks for the useful project.

      They'd probably get acceptance from basically everybody, because they're doing the work to solve their problem which doesn't restrict external developers. Maybe that's what they will do. So far, however, it seems like they want a bunch of information from the projects and complain about practices they dislike without wanting to put in the effort to fix anything.

  18. Glen Turner 666

    Comment on paper

    It's interesting that this proposal is a one-way street.

    If I buy a Google phone I cannot build my own copy of the complete source code, obtain the identical binary outputs, and so verify that Google's build systems have not been subverted.

    The paper's suggestion that there should be assertions about the software process is excellent -- are all commits signed from a hardware key with a SAK keypress -- that sort of thing. But again, why is this limited to open source software? If I am giving someone money for software surely that's more reason to be provided with assertions about the integrity of the source code and build system. At the moment all we get from commercial vendors is unverifiable bullshit -- right until their security issue SolarWinds was claiming that their software processes were best practice.

    There's also a substantial amount of cost-shifting in the proposal. Distributions, such as Red Hat Enterprise Linux or Google Android, have substantial income to fund housekeeping tasks such as backporting bug fixes. But the paper proposes to move this expensive task to free software authors, who often don't gain much income from the infrastructure library they authored. This is a task made more expensive by Google's reluctance to share with developers in a timely way its fixes to the open source packages Google uses.

    The notion of federated identities for software developers could very easily degenerate to Google authorising software developers. It won't be too long before the intelligence-security apparatus removes entire nationalities from the 'critical infrastructure trusted' community: they're unlikely to accept a Russian national. But this in turn means that Russian nationals discovering security issues will be pushed towards engagement with the malware firms surrounding the FSB.

    And again this is a one-way requirement. I cannot obtain the identities of each individual who has contributed to the code on my Google phone. Let alone attributes of those identities such as nationality, criminal record and prior employment. I am required to trust Google's hiring processes.

    In short the paper has some good ideas -- unsurprisingly since they are common in the discussion around SolarWinds -- but has well underbaked the development of those ideas into policies.

  19. Mike 137 Silver badge

    Don't make me larf!

    "Google is a founding member of the Linux Foundation's OpenSSF (Open Source Security Foundation)".

    "Open" and "security" don't ever seem to have applied to Android.

  20. Ben Tasker

    > Google, it turns out, does not entirely trust the usual open-source repositories and package managers. The company keeps "a private repo of all open source packages we use internally

    Trust aside, this is good practice.

    If your product relies on 3rd party software, you're putting all hopes of the future viability/buildability of your project on the repo for that project not going down.

    If it's something you need/care about, then you should be mirroring it and running builds/installs against your own mirror.

    The number of developers I've seen include "pip install" in internal build instructions in my career is simply maddening. I say maddening, but it's tended to be them that end up working late when "pip install widget" no longer works because "widget" is no longer available.

    Unless it's a nice-to-have rather than a must-have, maintain a production mirror.

    Of course, this doesn't work quite so well if you're using something like Node....

    1. Claptrap314 Silver badge

      This. By 2012, as a swe with no real operations experience, I became more & more concerned about what was really going on when I did a "bundle install". (That's ruby for "find and install all my dependencies".) In 2013, and at a late-stage startup, I raised the issue of cloning rubygems, or all least all of our dependencies, onto our build servers so that we would not have to deal with this problem.

      Of course, at the end of 2013, rubygems was taken down by the maintainers to contain and repair a hack.

      More interestingly, and much more seriously, four months later there was a semver failure that affected a huge swath of the ruby community. Since we had not learned our lesson, we were struggling some time.

      For the curious, the semver failure was as follows: A popular package depended on an obscure one. The dependency, however, involved directly including a particular file from the source. The obscure package did a refactoring (no API change) that eliminated the file in question from the source. By semver, this was a point change in the lower package. Since the popular package specified "latest" point change, bundler was insistent about using the new one. With no way to blacklist a version of the low level package, any fork, fix & self-hosting would have spread to a large number of packages...

      1. Ben Tasker

        > Since the popular package specified "latest" point change, bundler was insistent about using the new one.

        That reminds me of a vaguely related and slightly more recent one actually.

        CentOS 7 boxes ship with the 3.10 kernel. A kernel feature not present in 3.10 was required, so a more recent kernel was required.

        No drama: add the Elrepo repo and we can have the latest (for sake of example, lets say 4.9.17 as I can't remember the actual version). So orchestration was amended to add the repo and install kernel-ml.

        But, a little while later, a problem becomes evident.

        Elrepo do not archive older versions, and a new release (say 4.9.18) has been made. All new builds are using 4.9.18 (which, dun dun dun has a serious bug that affects us) and there's no way to install 4.9.17 from Elrepo because the packages are no longer physically there.

        This was solved, unsurprisingly, by adding a build of 4.9.17 into our own repos.

  21. ClosedJar

    There is nothing noble about Google. I have been watching this company grow through the years. They want to control everything around them. Open source is the last large road to expression and a sort of freedom. Who would not want to control that? Large companies like Apple, Microsoft and Google is all about control. Always speaking of the greater good like they are a paragons of virtues. Yes open source needs work but is up to the community to reassess its security and how they work. Ideas should be welcome but in the end is not Google that should be preaching to others. Remember Android? Nuff said!

  22. Henry Wertz 1 Gold badge

    Google can do this

    Google can suggest what they want. As people commented, I do not expect the libjpeg-turbo, libpng, ffmpeg, etc., developers to follow any of this -- especiallly ffmpeg due to the large amount of ongoing code changes ffmpeg always seems to have. To be honest, if google wants repos where 2 people must vet every change, and the repo owner has a verified identity, they can continue to run their own repos with a verified owner and have 2 people eyeball each patch. I in no way expect upstream to do this for them.

  23. martinusher Silver badge

    All humans have limits

    I've been programming for many years and one of the most lucrative sources of work that I've had are clever programmers. They're the sort that get so bound up in their work that they don't see the bigger picture and as a result often don't see the edge of the intellectual precipice until they've tumbled off it. I get brought in to handle the resulting train wreck. These people aren't evil or bad employees or whatever, they're actually essential to move everything forward and should be appreciated and rewarded as such, but one of the key skills in any software project is knowing how far you can push things before the design gets away from you. I learned this at a fairly early age -- I've still got a copy of one of my earliest assembler programs which 'one day' I'll get around to figuring out how it works (yes, it does work). So while its boring and a little constraining to have to do all the 'bureaucratic' things that we need to keep control of a design the penalty for not doing so is an ever increasing avalanche of bugs. The real skill here, though, is knowing how to chart the course -- knowing what's important to focus on, what can be discarded, how to actually go about solving the problem and building and testing the project, its like navigating through a gap between the brilliant 'almost works' and the bureaucratic 'it will be perfect if we ever finish it'.

    I think what Google is trying to do here is laudable. There are some critial libraries in the FOSS world that need to be managed and maintained properly because they are critical. Google could just write a propritary version of the code, an "I'm all right, Jack" move that really wouldn't be that smart, or they could add resources to solidify and maintain these components. I know if I was an author of one of these components I'd welcome the help (and, let's face it, a bit of cash might also be appreciated). In an ideal world these creators could go off and make something new.

    This brings up another concept worth thinking about. I've had personal experience of some (clever) programmer that's worked on the same piece of code for 14 years now. Its obviously been rewritten a few times over the years but its the same basic design and many of the structures and modules can be traced to the beginning (and I'm not even sure it was just 14 years ago). One signature feature of this code is that its soundly hated by anyone who's stuck working on it -- its now written in C++ which has made it even more obscure. What it does is actually quite straightforward, its a straightforward 'background loop and insterrupt service routine' piece of code, the sort of thing that you find at the core of, say, Arduino software. If the company had the resources and the will then they'd pull this fellow off this code and give him an exciting new project, leaving the 'maintainance' team free to rationalize it, switch over from older home-made libarary support to mnaufacturer's library support and so on. You can get too involved with just one piece of code. Its unhealthy -- you need to be able to walk away, hand it off and do something else.

  24. JavaJester

    Secured or Vetted Builds?

    The same thing could be accomplished without changing the OSS community. An entity, such as Google or some federation of companies, could essentially mirror the various critical open source projects. They could create the signed builds they want. They could review each change and include changes that passed whatever reviews they wish to do. They would be free to make pull requests with improvement or perhaps even become a committer on the project. The people who want/need these security assurances should be the ones to provide them. They should not burden the OSS software nor try to change their culture.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like