GitLab.com luckily found lost data on a staging server GitLab.com, the wannabe GitHub alternative that yesterday went down hard and reported data loss, has confirmed that some data is gone but that its services are now operational again. The incident did not result in Git repos disappearing. Which may be why the company's PR reps characterised the lost data as “peripheral metadata …
Maybe I'm too cynical here, I cannot rule this out, but in my opinion Gitlab didn't have a choice but to go public. For the simple reason of damage control.
Think about it: what do you think would have happened if they covered things up only to see the details leaked at a later time? Then it would become double trouble; not only would the community start criticizing them about their plain out ridiculous backup "strategy" as well as them trying to cover it all up. If they had gone this route and the details did eventually emerge then they could have definitely kissed their companies reputation goodbye, maybe even the entire company.
So I don't see any goodwill here, only simple damage control. BUT.. I may be a little overcritical.
Even so... Overlooking the fact that 100+Gb worth of data gets "archived" in files of a few kilobytes large has nothing to do with making a simple mistake, that is a plain out display of stupidity at its finest.
"Maybe I'm too cynical here, I cannot rule this out, but in my opinion Gitlab didn't have a choice but to go public. For the simple reason of damage control."
Maybe you are. All too often businesses come up with a different choice in that their responses are from a standard PR playbook: "Only a few....", "your ... is important to use" etc., none of which is believed by anyone with two brain cells to rub together.
If at the end of this they end up saying that only a few users were affected there'll be plenty of detail to give it credence.
"Maybe I'm too cynical here, I cannot rule this out, but in my opinion Gitlab didn't have a choice but to go public. For the simple reason of damage control."
Yes, they did the right thing whether it was intentional or unintentional. Covering up stuff ups never works and always makes the company look even worse (here's looking at you TalkTalk).
That said, this time GitLab really got lucky and they must significantly improve, and test, their back up capabilities.
You mean this? as posted by someone else yesterday:
Yesterday,
All those backups seemed a waste of pay.
Now my database has gone away.
Oh I believe in yesterday.
Suddenly,
There's not half the files there used to be,
And there's a milestone hanging over me
The system crashed so suddenly.
I pushed something wrong
What it was I could not say.
Now all my data's gone
and I long for yesterday-ay-ay-ay.
Yesterday,
The need for back-ups seemed so far away.
I knew my data was all here to stay
Now I believe in yesterday.
Shall I compare thee to a proper backup?
Thou art more lonely and more disparate.
The tape rewinds as I lift my coffee cup,
And GitLab’s data hath all too short a fate.
Sometime too hot the scripts of admin shines,
And often is his gold complexion dimmed;
And every fair from fair sometime declines,
By chance, or nature’s changing course, untrimmed;
But thy eternal data shall not fade,
Nor lose possession of that fair thou ow’st,
Nor shall cockups brag thou wand’rest in his shade,
When in eternal lines to Time thou grow’st.
So long as Git can clone, or eyes can see,
So long lives this, and this gives life to thee.
~ William Websphere
This post has been deleted by its author
"±4979 (so ±5000) comments lost"
What does that mean? Does it mean they may have either lost or gained 4979(/5000) comments. It would be interesting to see the comments that were gaines - something like "I'm sorry Dave, I can't recover any more data for you as you appear to have no viable backups", "Sorry Dave, I can't allow you to do that"
I guessed what they were trying to imply but the ± doesn't mean that when placed before a number. It was more a precursor to set up the Space Odyssey/HAL reference rather than an actual question.
Sorry the Register font doesn't support the Rhetorical Question Mark.
At least they were honest, transparent, and were open to people looking at what they were doing or going through.
We've all been stung by outages in the past and have never quite known why it went down, or why it took so long to come back up.
GitLab should be applauded. I was affected by this yesterday, but I'm happier now they've had this issue and shown what they're like in a crisis.
A pint for the team. Just the one, I'm not made of money.
.. as with all disasters, the upshot is that it will now follow sane processes for a while. Most businesses follow this up and down curve when surviving a brush with disaster (or an actual disaster) results in a correction, and it will take a while again before the company sinks back into the sort of complacency that makes it again drift towards problems.
So, in a few weeks, GitHub will be the best place to be :)
"The Register imagines many developers may not be entirely happy with those data types being considered peripheral to their efforts."
The Register (or rather, Simon Sharwood) imagines wrong. It just so happens that both my organisation and I personally, have GitLab accounts that we use to coordinate development of a few tools which are, after all, critical to our business. We use free accounts for this and our repositories are private (an option not available with their main and better funded competitor).
Their issue tracker is how we demonstrate compliance for certain aspects of our quality management system.
Are we unhappy? Not at all. We get a lot out of it and pay (yet¹) nothing in return, so hardly in a position to complain. Apart from that, we take regular backups of our projects (that is, the issues, wikis, attachments, all the stuff ancillary to the actual git repos) that we use to recover lost data in a number of ways. If it came to it, we could even self-host a GitLab server and dump our data into it, but we'd rather spend our efforts in doing what we're good at.
In short, shit happens, smart people (and even us) plan for it. Sometimes plans go wrong. Occasionally the holes in the Swiss cheese line up. Too bad. That doesn't mean that anyone is incompetent or an idiot. It's just risk management in action.
I hope someone more analytically inclined or with a bit more of a technical background will blog about this here in The Register, so that we too can learn lessons from this incident.
If there is one thing I am "not entirely happy with" however, it's the "more sensationalism and less substance" turn that The Register has taken in the last few years and which just seems to be getting worse, which is why I hardly bother reading it any longer.
¹ We intend to, started off with a free account to check them out, then things got really busy and we haven't got around to that yet. Sorry about that.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
