Microsoft's GitHub under fire for DDoSing crucial open source project website This month you may have noticed the servers used by the GMP project – an open source arithmetic library at the heart of GCC and other programs – slowed to a crawl. It was due to a deluge of network traffic, the source of which is quite surprising. The packets appeared to come from servers associated with Microsoft. Torbjörn …
I think they probably would if it was all internal, but they didn't bother to cache for users of GitHub. In fact, the users probably prefer that, because it means there's no risk of a faulty cache causing problems for their automation. The people who end up being the target of a lot of forks, none of which are caching, have a very different view of that. Of course, each fork could and should have cached one copy of the source which could be duplicated locally for each different test environment, which would have cut the bandwidth requirements by a couple orders of magnitude. Users probably didn't bother doing that because it's all GitHub's incoming bandwidth and they don't run the server being hammered, so they don't think they have to care. Caching across forks would probably be more difficult without having Microsoft make assumptions for all users of the platform.
Why are builds like this continually grabbing from *any* server?
> This build was configured to run parallel simultaneous tests on 100 different types of computers/architectures
And all of these architectures need their own copy of the GMP code? Sure they aren't just using the same sources with the usual flurry of #ifdef's?
For that matter, what about the rest of the ffmpeg and dependencies source code? Are these being grabbed afresh for every single architecture? That certainly seems appears to be the case, if I'm reading the updated script correctly[1].
The DDOS aspect not only appears to be built in, but why aren't all these copycat builders pointing out that it is wasting *their* resources as well (time, bandwidth and electricity)?[2]
[1] and the author of that update wants to have a careful think - any guesses whether running a fetch from *any* version control server - in this case, a Mercury server - is going to more costly for both ends than a wget of a single tarball from an FTP server? Or could a version control client perhaps pull down *just* recent changes - oops, no, we deleted the local copy, can't do that!
[2] oh, it is just Github's bandwidth and electricity? Well, waste away!
>> And all of these architectures need their own copy of the GMP code? Sure they aren't just using the same sources with the usual flurry of #ifdef's?
This is what happens when every test spins up a new VM. It's appallingly inefficient, but a lot of people now think it's "the way you do it". I don't know where this is coming from but I suspect it's just fashion.
Yeah exactly, that's how kids think nowadays, its not enough cut and paste a module or copy a few directories you've got to fork the whole godammed machine for a one line change. Its not fashion its a combination of ignorance, ineptness and laziness.
But I suppose this is how things go, the abstractions get increasingly coarse grained so that you can work on bigger and bigger problems and we forget about the complicated stuff going on deeper down. There's an xkcd for that somewhere.
There are benefits to every build starting out from a known clean system. Before I migrated all of my projects to docker build environments I would frequently come back to a project from 6 months ago and find it no longer builds because of some change I made to my dev machine to accommodate another project...
> I would frequently come back to a project from 6 months ago a
There is a teeny, tiny, itty bitty difference between cleaning out every six months and starting from scratch every time a batch file enters a loop to trigger another build.
And complain to whoever wrote the stuff that is changing the environment (or requires you to change it) in a way that can affect another project: that is just bad work on their part. You really should not need to have to resort to Docker or VMs or anything else to keep a working environment. I know a lot of projects piddle about like that, but maybe we can get them to grow up one day.
This post has been deleted by its author
It wouldn't be good, and we wouldn't be happy that they'd done it. However, it would probably happen, intentionally or not, as the amount you sold increased. Especially if you're making money from this, it's worth knowing how much of that resource is in use and whether you need to do something to increase that resource.
In this case, the users of the GitHub actions should really have cached data instead of downloading it so many times, and had they done it, the server wouldn't be so overloaded. However, even without that, a project which continues to be popular will reach a point where a single 1 Gbps link is no longer enough bandwidth for all the people making requests. They're providing the server for free, so I'm happy to let the users just have to deal with really slow downloads and they can try to help improve the system if it gets slow enough. Still, that's just my view on how they can deal with the resource exhaustion, not a way to prevent the resource from being exhausted in the first place which will happen eventually even without the GitHub forks spiking.
>” Periodically check if your "server grade" infrastructure still meets modern standards.”
WTF does that mean in this context?
About the only thing that is relevant, is ensure you have put in place limiters so that you don’t get hit with stupid bills resulting from third-parties overloading your servers.
They're running the infrastructure, but if a GitHub user writes a script that downloads a file from some server and then tells GitHub to run that script hundreds of times in parallel, I don't see how Microsoft is the one to blame here, any more than if someone did that on AWS or some other cloud provider. They do have a responsibility to monitor for malicious usage of their service, but this case isn't clear cut in that regard.
The project in question, on the other hand, can and should do better than just randomizing the cron job's timing. It should be making use of the caching that GitHub actions provides.
> maintaining an attractive nuisance
Loving your characterisation of Github users as having the awareness of seven year olds. That does strike at the heart of the problem for this ffmpeg builds guy (including his response of "you should change the cron timings".
Likewise if someone uses Amazon web services and leaves their buckets unsecured. The computer with the unsecured bucket is Amazon's, hence it's Amazon's fault.
Likewise if someone posts abuse on Twitter. The computer sharing that abuse it Twitter's, hence it's Twitter's fault.
Likewise if someone is hit by a Ford Mondeo. The vehicle doing the hitting is a Ford, hence it's Ford's fault.
I can't see a problem with any of this logic!
Your making false comparisons, except perhaps with Twitter. GitHub is a service that is run by Microsoft. This makes Microsoft liable for what happens on it. This is also why the T&Cs are important because they indemnify Microsoft for anything that happens on the platform, ie. users can be made to pay for any costs incurred.
In some jurisdictions, Twitter doesn't benefit from the US Safe Harbour provisions and can and, indeed, has been made liable for posts on the platform.
You obviously dont understand how build tools work. Microsoft does not provide this tool, the tool does whatever it likes in this case pull bytes from the GMP repo. Microsoft at no stage has not modified or provided any significant library or tool in the chain of the b uild.
I wonder why the ffmpeg-build maintainer simply won't create a mirror of the gmp repository on github and clone from there inside his scripts. Oops things happen, so how about just cloning the repo for automated mirroring (which I assume is a lot less aggressive) and patching the scripts to use the mirror?
He was managing to run a Mercurial client to get the sources, which rather implies he could have been using it properly (just checking for updates instead of explicitly chucking away the local copy every time the build run). Even pushed from his local copy into a git repo and made the script use that (hence all the copies of the script...).
Saying "Github doesn't do Mercurial" - well, it is true that Github doesn't provide all the tools to work with a Mercurial repo in the same way as it does a git repo, but this guy already knows enough to use a command line Mercurial client to pull the library down, it can't be beyond him to add in use of the git command line tools as well.
Change the returned data to perform a dangerous action on anyone running that workflow.
Problem solved.
This was always the problem with "let's just include this code from the web in our webpage/scripts" as if that was a valid, never-down and free resources for all. Remember the fuss when certain crucial Javascript includes went offline and thousands of websites "broke" (more accurately: fell over because the crutch they'd been using free of charge disappeared).
It certainly shouldn't be pulling hundreds of copies of the code just for a test suite.
I have responses back to those IP ranges being a 0Kb zip file, at minimum, but more likely a zipbomb, malicious code, stroppy messages that means their test suites all fail, etc.
They'd soon "fix" it then.
But it sounds like the "IP ranges" are just "anyone using GitHub", so you'll be delivering that zip-bomb to all sorts of folks who just wanted one, legitimate, fetch from your repo. Not only is that harsh, it is probably considered malicious in some jurisdictions.
A zip file containing a short readme file explaining why this range is currently receiving special treatment might be more effective in the long run.
You seem to be conflating embedded code with build tools. From the project description GNU MP is a library for arbitrary precision arithmetic, operating on signed integers, rational numbers, and floating point numbers. Thus, it exists to be used by other code and the authors don't have a problem with that. Most unixes have package managers that allow GMP to be installed but they also usually have their own mirrors of the relevant code.
What is causing the problem is that myriad forks of the ffmpeg repo have cloned not only the code but also the workflow and these, in turn, are kicking off an awful lot of clone requests of the GMP repository. In many CI setups caching is standard and, indeed pretty much a requirment for anything using images from Docker hub. GitHub doesn't seem to have this and it also doesn't seem to have any kind of rate limiting. In other words, it's been very poorly set up. But the T&Cs mean that developers have agreed to indemnify GitHub/Microsoft for any damages incurred during the use of the platform…
> In many CI setups caching is standard and, indeed pretty much a requirement for anything using images from Docker hub. GitHub doesn't seem to have this and it also doesn't seem to have any kind of rate limiting
True about the lack of rate limiting.
But whether or not Github provides caching would not be relevant in this case - if you read the (even updated) build script that the article points at, the guy was explicitly deleting files and then fetching them again using a command line Mercurial client, which he replaced with explicit use of a wget to an FTP site. Those would bypass any Github caching.
The 100s of architectures refers to the ffmpeg builds, not specifically GMP.
It *may* be the case that the only reason ffmpeg "needs" to be rebuilt for each architecture is only because that allows the GMP build to optimise itself, but that seems unlikely. From GMP's point of view, many of these "architectures" may well be the same, just some other part of ffmpeg wants to know the difference (e.g. forget about x86 variants, where does this OS variant store this file?).
"Oh, but why don't they just host their project on github!" (the question that would first pop into many heads and admittedly, would solve that problem at least in the short term)
Because they don't control it. Some would rather host their projects themselves, git or otherwise. I'm afraid I'd be in that camp too.
Even at my level, I'm sitting here with my thumb up my ass today. All the things I was going to get done aren't, because github is down and I can't pull the sources I need.
> "Oh, but why don't they just host their project on github!"
Microsoft just *love* people who complain like that!
> Some would rather host their projects themselves
Or even just use another third-party project hosting site.
Come on World, what is wrong with you, *every* project should be on Github! What Is Good For Microsoft Is Good For America etc etc.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2023
