Total Inability To Service User Pulls: GitHub wobbles with a good old Thursday TITSUP

Source shack GitHub has taken a tumble today with many users finding pretty much all of its services either degraded or borked beyond belief. Problems appeared to start at around 1400 UTC, judging by the shrieking on social media, with GitHub admitting that, yup, something was amiss with the API and webhooks (required for …

[Anonymous Coward]

If you store your project code on an online repository...

..and don't have an up to date local backup then you deserve everything you get when this sort of thing happens. No sympathy whatsoever.

Whatever the marketing lizards say, there is no such thing as a 24/7/365 service anywhere.

[NullNix]

Re: If you store your project code on an online repository...

If you store your source code in an online repository, don't have an up to date local backup, *and are using git*, I don't know *what* you're doing, because all your local copies must be shallow clones -- are you *that* short of disk space?

It takes real effort to avoid having a local backup with git (which is why github is more or less dispensable to many of us oldtimers who prefer email: git gives you all the code hosting stuff in every local repo :) )

[MatthewSt]

Re: If you store your project code on an online repository...

Seconded! All of our devs have actually got a 2nd "remote" configured to a fileserver for precisely this scenario. Granted we still build in the cloud, but it doesn't stop us collaborating.

[JimBob01]

Re: If you store your project code on an online repository...

“Whatever the marketing lizards say, there is no such thing as a 24/7/365 service anywhere.”

Who would want a service that is guaranteed to be offline one day every 4 years? I’m quite happy with plain old 24/7.

[Claptrap314]

Re: If you store your project code on an online repository...

I was a Google SRE. There was a service there whose reliability was too good. They deliberately took it down every now and then (for a minute or so) to ensure that no service that used it required it to be up all the time.

Wise policy.

[Anonymous Coward]

Re: If you store your project code on an online repository...

This policy gets a mention in the SRE book. Agree, that's smart.

[vtcodger]

Re: If you store your project code on an online repository...

I don't use Github because I don't need it. And for the most part it's too complicated for my aging mind. But my understanding is that Linus Torvalds wrote git to handle collaborative efforts that were too complex for RCS, Subversion, et. al. That kind of suggests to me that there are times when some folks really need the "official" version of code, not their local copy/copies.

[W. Anderson]

Now Windows based?

Since Microsoft purchased and now controls Github, did the company switch over it's infrastructure technology to Windows Server?

This is likely the cause if such switch took place, as did happen with Hotmail after it was purchased by Microsoft from Yahoo, which ran the e-mail service for years on FreeBSD without problems.

[Anonymous Coward]

Re: Now Windows based?

"as did happen with Hotmail after it was purchased by Microsoft from Yahoo, which ran the e-mail service for years on FreeBSD without problems."

HoTMaiL had nothing to do with Yahoo at any point.

FreeBSD was only used for the web front-end, Solaris provided the actual email grunt works.

[Anonymous Coward]

Re: Now Windows based?

Please, the man had a good rant going. Don't ruin it by bringing up facts. This is the internet - a facts free zone!

[bigmacbear]

Re: Now Windows based?

"Germans?"

"Forget it, he's rolling."

[bombastic bob]

Re: Now Windows based?

regardless of detail, switching to Windows broke things in the past (not surprisingly).

I wonder how much of that breakage was due to case sensitive file names...

Of course NOW we have github, which is (no doubt) FILLED with case-sensitive file names.

My guess is Linux is run by the cloud instances. That would be consistent with the original development of git itself. And Azure should be able to handle that, right?

[Will Godfrey]

Hmmm

Last pull I did was yesterday, no probs then. Today I've just been pootling around on my local master.

P.S. Also wasted a lot of time sorting out an intermittent network fault. Turned out to be a dodgy IDC in a switch {grrrr}

[Claptrap314]

I thought folks here were clued in...

The issue isn't the git, it's the hub. With web hooks down

none of the CI/CD pipelines relying on them work

none of the chat notifications for commits work

none of the chat notifications for bug reports work

...

Programming works fine. It's the other stuff which was down.

[Claptrap314]

FREE SERVICE! Get your FREE SERVICE!

Any business relying on a free service for business-critical processes deserves to go down in flames.

I doubt that Github has gotten very far down the road to moving to Azure yet. The best tell is how it fairs during the next global failure that Azure has, currently scheduled for sometime next month.

[bombastic bob]

Re: FREE SERVICE! Get your FREE SERVICE!

I've worked with private github repos for 2 different companies. It's not free. But it's not expensive, either.

[GrahamRJ]

There are alternatives

I'm using GitLab at work. GitHub was asking a bit too much for hosting non-open-source projects, whereas GitLab doesn't care. And their servers have never missed a beat. All my DevOps stuff happens on an in-house server, so all they need to do is handle pushing/pulling the repos. Some day I may move that up to the cloud, but where I am at the moment, even running Jenkins and Bugzilla was a dangerously modern idea when I arrived. (Seriously, an engineering company without an issue-tracking database, in 2018. My conversations could be roughly summarised as "shoot me now, or give me a bloody server".)