back to article GitLab deploys on a Friday and ... is down for a few hours

GitLab, a hosted git service not unlike Microsoft's GitHub, was down for some users as of Friday morning, Pacific Time. Around 1634 UTC (0934 PT), the code hosting service started returning 503 Service Unavailable errors to those attempting to access the website. Software developers who depend on the service were quick to …

  1. Throatwarbler Mangrove Silver badge

    Not surprising

    If the configuration file for the hosted version is anything like on-prem, it makes seem simple and readable by comparison.

    1. Mostly Irrelevant

      Re: Not surprising

      It's almost assuredly worse. It's probably deployed by a continuous integration system that's putting together the config from multiple places config/secrets/variables every time.

  2. Claptrap314 Silver badge

    Prod change on Friday? No plans for the weekend, then.

    It's NOT a superstition. Fat fingers & fat brains were the cause of 80% of OMGs when I was at Google. (2015-2016)

    Shipping prod on Friday that isn't a critical fix is flat arrogance.

    Serves them right.

    What I can have that they cannot ------->

    1. chivo243 Silver badge

      Re: Prod change on Friday? No plans for the weekend, then.

      Pffft... Rank amateurs, never, never schedule any production anything on a Friday! No updates, no patches, no upgrades, no migrations, just no! Use Fridays for cleaning up tickets and finishing old business, ease into the weekend ;-} No manager in his right mind wants to call out $upport on a Saturday, because Friday's actions went pear shaped... Wednesday was always my target day... one day to fix it yourself, then another day to call for help!

    2. xyz Silver badge

      Re: Prod change on Friday? No plans for the weekend, then.

      I smell devOps with a whiff of Agile.

      No sysAdmin would ever break the "no deploy on a Friday" rule. It's a commandment if I remember fish, drink beer, don't deploy.

  3. ChoHag Silver badge

    It's not a superstition, it's a rule.

    And here it's used as a sign that GitLab's sysadmins are stupid (never deploy on a Friday: it will go wrong and YOU will lose a weekend fixing it) or feckless (tell your manager that) or both.

    1. Joe W Silver badge

      It is also in the SOP of the BOFH, it is written that there must be no changes cuspid to a diurnal period of absence or somesuch...

  4. abentley

    Actual cause: old Terraform config accidentally deployed

    They weren't really blaming GCE, that was just the location of the problem. The root cause was "Part of a change request, an old pipeline was triggered, applying an obsolete Terraform plan to the Production environment."

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like