GitLab deploys on a Friday and ... is down for a few hours

GitLab, a hosted git service not unlike Microsoft's GitHub, was down for some users as of Friday morning, Pacific Time. Around 1634 UTC (0934 PT), the code hosting service started returning 503 Service Unavailable errors to those attempting to access the website. Software developers who depend on the service were quick to …

[Throatwarbler Mangrove]

Not surprising

If the configuration file for the hosted version is anything like on-prem, it makes sendmail.cf seem simple and readable by comparison.

[Mostly Irrelevant]

Re: Not surprising

It's almost assuredly worse. It's probably deployed by a continuous integration system that's putting together the config from multiple places config/secrets/variables every time.

[Claptrap314]

Prod change on Friday? No plans for the weekend, then.

It's NOT a superstition. Fat fingers & fat brains were the cause of 80% of OMGs when I was at Google. (2015-2016)

Shipping prod on Friday that isn't a critical fix is flat arrogance.

Serves them right.

What I can have that they cannot ------->

[chivo243]

Re: Prod change on Friday? No plans for the weekend, then.

Pffft... Rank amateurs, never, never schedule any production anything on a Friday! No updates, no patches, no upgrades, no migrations, just no! Use Fridays for cleaning up tickets and finishing old business, ease into the weekend ;-} No manager in his right mind wants to call out $upport on a Saturday, because Friday's actions went pear shaped... Wednesday was always my target day... one day to fix it yourself, then another day to call for help!

[xyz]

Re: Prod change on Friday? No plans for the weekend, then.

I smell devOps with a whiff of Agile.

No sysAdmin would ever break the "no deploy on a Friday" rule. It's a commandment if I remember correctly...eat fish, drink beer, don't deploy.

[ChoHag]

It's not a superstition, it's a rule.

And here it's used as a sign that GitLab's sysadmins are stupid (never deploy on a Friday: it will go wrong and YOU will lose a weekend fixing it) or feckless (tell your manager that) or both.

[Joe W]

It is also in the SOP of the BOFH, it is written that there must be no changes cuspid to a diurnal period of absence or somesuch...

[abentley]

Actual cause: old Terraform config accidentally deployed

They weren't really blaming GCE, that was just the location of the problem. The root cause was "Part of a change request, an old pipeline was triggered, applying an obsolete Terraform plan to the Production environment." https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15999