What a massive fail
And running a script seems like a bandage on a bandage. The dependency on outside packages, libraries, and blindly copied code has really gotten out of hand in recent years.
Coming from the SysAdmin world before I started Dev, hosting the repos internally makes no difference for several reasons:
1. You still have to trust the upstream repository at some point, so unless you're planning on writing every library yourself the risk still exists.
2. The rate at which bug and security fixes come into public packages make it just as much a risk to host an internal mirror and have to wait for "approval" to update those packages meaning we're back to the original issue - write vulnerable code waiting to allow for package updates on the internal mirror or write your own libraries..... which also isn't always a possibility depending on the vendor you work with.
3. In the case of Rubygems at least, ALWAYS EXPLICITLY specify where to download the private package from rather than assuming that Rubygems/NPM/PyPI/etc will figure it out for you.
What you suggest requires more time and effort than most companies have available. I know my team doesn't have time to maintain that, nor do our IT or (internal) Infrastructure teams.
Additionally, the primary issue present in the findings from the hacks revolved around conflicting/duplicate names used in private packages that match public packages and using the laziness of a team where they don't explicitly define where to get their private packages from. Your solution doesn't solve that problem, if anything it just leads to dev teams relying more on local dev environments than on the build system. There are hundreds of tools that exist now to allow a developer to setup their own dev environment without EVER needing admin access to their local system.
Finally, if you really think that companies that currently operate simply by giving their developers business requirements and let those teams make their own determinations what tools, libraries, languages, etc to use to meet those requirements. Those organizations would lose a ton of developers as we do not like being told HOW we have to do our job.
Okay, I'll bite. What proposal do you have to ensure that, say, a ruby shop is able to continue to do its builds the next time that rubygems gets taken down? When there is another semver failure?
Devs must be able to explore. That requirement does not imply that the build process should be vulnerable to nonsense.
Yes, there is a cost to everything. And I specifically called out that smaller shops might not be able to do a full review of every source that they use. But not even having the source? Irresponsible.
I work in C# .net C++ C and yes, we sometimes use external opensource libs nowadays most likely got through NuGet. If one is needed the dev will download and code with it if it does what they want and is approved*, it will be added to source control. From then on until it needs to be updated that is the version that is used in the build process. What is shipped is every file that is needed and an installer of some sort.
Because we control when a new version is used, we can test it also any bugs that are found the changed lib is a place to investigate.
This approach worked well for us in the past, but now we have open-source audit software Black Duck we now get the constant nagging of not using the latest version.
* No opensource code can be used unless it has been approved by the aptly named opensource committee.
With python, its dead simple - you push your private packages to a private pypi repository (or pay someone to host it for you, eg artifactory), and configure pip to install preferentially from that repository, with the private repository falling back to the public pypi only when the package is not found locally.
Here is the stack overflow Q answering that :D
That's exactly how the companies got hacked. Developers and/or build systems were tricked into using an impostor library from a public source. All it took was bumping up the version number in the impostor so the "better" one was chosen.
I know it's a hassle to maintain a private repo of public code, but that's what's needed for security.
And for reliability. Your tests were performed using some version of imported code. If you rebuild using another version, those tests are invalid. Just because you're using a later version with supposed bugfixes doesn't mean that there aren't new bugs, or that the fixes don't trigger latent bugs in your code.
You need to know when you import a new version, and repeat your testing when it happens. Until that point your code is made less reliable by importing a bugfix, not more reliable.
Mmm, I have to disagree, at least for python/pypi/artifactory.
If I have published a package mycorp-foo to artifactory, and I ask artifactory for mycorp-foo, artifactory will respond with the versions that are published on artifactory. It won't say anything about any mycorp-foo packages that are published to pypi, since the package exists locally.
There's still a bunch of issues involved with that, think NPM's left pad. What a number of places do is proxy ALL requests through the internal archive. That way you keep a copy of public libraries so you maintain buisiness continuity and avoid left pad type issues.
The problem is that it does not protect from thos sort of attack as you're still querrying one repo and because most dev's don't want to deal with security issues they will just set the deps to be the latest version, or the latest within a specific release eg 2.x is fine.
"The problem is that it does not protect from thos sort of attack as you're still querrying one repo and because most dev's don't want to deal with security issues they will just set the deps to be the latest version, or the latest within a specific release eg 2.x is fine."
I don't know what devs you work with, but in my experience in both FOSS and proprietary software, we don't ever set our libraries to use "latest" for any reason. Not even containers.
As for using the "latest within a specific release, eg 2.x" - that's the entire point of SemVer. Major versions are breaking changes, minor are new features, patch are bug/security fixes. I can't speak to other environments, but the rate at which we receive security alerts from upstream libs, we fix them by updating or we are vulnerable. There's risk either way, unless you build everything yourself.
It's easy you do this, this, this and this and twidlle this setting over there and its all good. Unfortunately most devs are to lazy or ignorant of consequences. this should all be default and the Devs should have no option because they can't change the settings because they are not not a local admin.......
Bad people rely on good people being lazy and ignorant and make a lot of money from that.
Bad people rely on good people being lazy and ignorant and make a lot of money from that. .... Lorribot
If that be true, Lorribot, I wouldn't like to be relying on good people remaining forever lazy and ignorant/apathetic and undereducated, whenever nowadays it is so easy to be kept busy and to learn so much more than was ever imagined possible before, about virtually anything and everything.... with the secrets and tricks that deliver something from nothing exposed for exploitation and employment/edutainment and enjoyment.
And you have World Wide Webs to thank for that, both deep and dark, heavenly and enlightened.
And some systems are driving themselves bonkers and to rapid catastrophic destruction doing what little they only can do, failing to stop the information and intelligence being more generally widely known and universally available.
I'd like to know how anyone would stop devs setting up their local envs. They'd have to have no write capabilities on their systems. I don't install system-level packages for development anymore - most of them are too old for the business requirements. So user-level tools like ASDF are used to pull down (in my team's case) multiple versions of python, ruby, golang, node, etc into our user space not at the system level.
Meaning there is no way to prevent me from doing it even on the most secure system without taking away my ability to do development at all.
.... and here we are again.... if you don't know who the source is, you can't trust what comes from that source. People think encryption is all they need to be secure - but without authentication you are not.
With proper authentication reliable sources can be pinned and accepted - and any attempt to spoof the source would become evident.
Just make sure the data is digitally signed twice and the connection encrypted.
Firstly, by the legitimate upstream developer, each individual file must be checked for the presence of a signature or the files would be considered tampered with. Pinning could transparently occur upon the first installation of any given package and can be self-signed. This would also allow for tamper-proofing of the application itself post-install. For reference, Microsoft encouraged this practice since Windows 98 with Authenticode. It's embarrassing that modern solutions don't incorporate the basics.
Secondly, the package archive should be signed by the repository owner, so that files can be safely mirrored without risk. Certificates should be manually added to the keychain, just like the situation with RPM/DEB packages on Linux distributions. This has been common for well over a decade in the Linux community, so there's no excuse not to implement and enforce this.
Thirdly, all connections should be TLS-secured with a trusted certificate.
At this point, the developer's legitimate source would have to be poisoned, as well as the trusted repository accepting the changes. Companies like Red Hat, Google and Microsoft could work together to provide a vetted set of releases and charge third parties a very tiny fee for the privilege of access... everyone would be happy.
This is just modern coding practice, convenience is very high priority while security is something that just gets thought about later - the software management bosses tell the coders that the app's can always be updated later so it's not necessary to check everything now.
I just checked my phone - it's OK, only five apps need updating this morning, it was ten yesterday.
Of course there has been - it works.
And, on top of that, you can reap extensive rewards by getting automatically inserted into all of that company's customer's networks as well.
It's a blackhat dream come true.
All that because, for the past twenty years, developers have taken the habit of not bothering to check what they're putting on their production servers.
I don't care how much you trust that Github repo, you do not put code on production servers that has not been vetted.
Once again, lessons are going to be learned the hard way.
Vetted by whom and how? The dependency chain in many of these cases is so large that the business would never get anything to market waiting for all the code to be vetted. And by the time it was vetted, there'd be a whole new slew of security patches and bug fixes to vet or risk putting............. wait for it.......... vulnerable code in production.
By the numbers, again...
1) The build system never has access to the internet. It reads source from a file server completely under the control of your company. Anything else, and you are adding a needless availability risk to your build process. Nevermind the security issue.
2) The server hosting the source files used by the build server does not run a true mirror of external sources. What the build server sees when it asks for the list of versions of package foo are precisely what the business has decided it should see. Did someone have a hissy fit and delete a package from the public sources? Goody for them, but you keep your copy, thank-you very much. Did someone break semver with their release? You've got that release, but you block it until you decide what to do about it. Has there been a point version update? You have it, but your build system cannot see it until you've done a review. Is there a version out there where an actively exploited CVE is out and the fix breaks your stuff? If the exploit actually cannot be realized against your installation, we can enable the "bad" version and keep running while making the changes to support the fix.
EVEN IF your company is unwilling to spend the money to review all of the point versions, if your system has the capability, then at least you have the ability retroactively to block bad stuff and to keep what you need to stay in business.
Notice that I've not mentioned anything about what the devs have access to. Indeed, it is the job of the dev to check out the New Shiny, even if it *gasp* involves Somebody Else's Code. Getting approval to put the New Shiny into the build process, however, should require sanity checking.
Biting the hand that feeds IT © 1998–2022