Splunk: Why we dumped Perforce for Atlassian's Bitbucket of Gits

Kurt Chase, director of engineering services at Splunk, is effusive about Atlassian's Git-based Bitbucket but doesn't have a lot of nice things to say about jilted source control system Perforce. As it turns out, Perforce has a rather jaundiced view of Git as a way to manage software. But more on that later. Two years ago, …

[Tom 64]

Saucy!

I've used both perforce and git, and if the decision between the two ever falls to me, I'll choose git every time. Perforce is little better than svn in that it can lock files (WHY?), merge conflicts are a bitch to fix AND you have to pay for it!

The time you save by not having to wrestle with p4 is time better spent with a pint at the end of day.

[bombastic bob]

Re: Saucy!

my experiences with "all of the above" suggests that p4 doesn't really give you any advantages in any way, and of course with svn vs git they seem similar. except, of course, git seems to "cloud" well and so public projects on github [and even occasional private ones] make a lot of sense.

one thing I really liked about p4, back in the day, was its support of FreeBSD [though it was binary-only and sometimes required that I install binary compatibility for earlier kernels]

[Paul Shirley]

Re: Saucy!

The only thing perforce supposedly does better is binary blobs. Better in the sense that it recognises they shouldn't be differed I suppose and that's led it to be the tool of choice in games dev.

Frankly it's just evil and regularly causes lost days fixing the workspaces when it fucks up.

[networkboy]

Re: Saucy!

I don't know that I'd dog P4 that bad.

I used to work at Intel and we converted over from ClearCase to Perforce. While the transition was a bit painful (as one would expect) the overall workflow was much better. Again, the big selling point for P4 was the blob handling.

Git may be better in many ways, but P4 has strengths where siloing of access to source trees and large teams are concerned too.

[Richard 12]

Re: Saucy!

If the only choice is ClearCase v Perforce, then yes, Perforce.

But that says more about ClearCase than Perforce...

[Anonymous Coward]

Re: Saucy!

" time better spent with a pint at the end of day."

Or if you work for Splunk, during the day as well. Standard equipment for all their offices is a one kegerator for every 100 employees plus a well-stocked liquor closet. Hell, they even had pint glasses and other materials that read "Splunk> Rhymes with Drunk"

Anon because I am a former employee. I loved my time working there...

[Richard 12]

There's a lot of thing Perforce just Won't Do

Like "Give me the full list of commit descriptions between labels X (new) and Y (old)"

You know, a core function of any source control system.

It's simple to get that for the current branch/stream, but actually impossible to get it to reliable include all merged branches/streams - you often get bits from before Y, and sometimes everything that has ever happened ever.

After four months of trying I finally got something that almost works, but not quite.

It means the automated build reports are rarely correct - and that is very scary.

In short, Perforce is a very poor tool for management of code history.

On top of that, it can't deal with line endings in a sane way, it can't show you the history of what got imported from other projects (it's apparently there but impossible to see in any way)... the list goes on.

In short, Perforce is a very poor tool.

I would say that git is at least a decade ahead, and that Perforce will never catch up.

[macjules]

Re: There's a lot of thing Perforce just Won't Do

There is that. I still prefer GitHub with Webhook integration into Jira, Slack and Confluence to actually using Bitbucket, despite Bitbucket's unlimited repository size limits. I also find GitFlow much easier on GitHub which shouldn't actually make much difference but it means that I can do an end of sprint report with a diff between 2 tags in just under 20 seconds using GitHub.

[Mike Pellatt]

Oh, what a surprise.

A proprietary software vendor trots out the hoary old line that F/OSS is, per se, costlier in TCO terms.

Not if the vendor decides to walk away from the product, or goes bankrupt, it isn't. Risk analysis & management and all that.

[Anonymous Coward]

A proprietary software vendor trots out the hoary old line that F/OSS is, per se, costlier in TCO terms.

.. without, I note, any data or influencing factors to back that claim up - if you make a claim about something you can measure you should better be ready with the metrics to determine that independently or risk being exposed as being a g.., no, wait ..

As far as I can tell from the story, their switch to Git based tech must have dropped TCO substantially.

Well done - also for the article. I prefer reality rather than regurgitating marketing brochures.

[DrXym]

Doesn't surprise me

These days decentralized VCS is *normally* the way to go and Git is the defacto choice here. If I were setting up any software project I'd choose git first unless there were a reason not to. I ported a very large project from CVS to Git a few years ago - trivial merges and updates went from taking 20 minutes down to seconds. Even compared to Subversion it's fast and stuff like merging / rebasing is just so much easier. One interesting thing about git is that the entire cloned repository with all its history often takes up less space on disk than a single Subversion snapshot because svn preserves a pristine copy of every file you checkout to save itself a roundtrip when looking for changes.

The main reason I might choose a centralized system would be a repo with millions of files, lots of changes, lots of binary data, Word docs, CAD files etc. Most centralized repos allow people to snapshot just a bit of it in an ad hoc fashion. Git can do a shallow clone but generally treats the repo as an atomic entity which is cloned in its entirety or not. Git can only workaround this with submodules or subtrees and certainly not in an ad hoc fashion.

[Anonymous Coward]

Re: Doesn't surprise me

Google are a notable example of this. They store everything, literally everything*, in a single repo that started life as a p4 repo. These days it's built on homebrewed tech, but looks/feels similar. They do it for a different reason though; they think a monorepo all but eliminates dependency and versioning overheads.

*Billions of lines of code, documentation, config etc for very nearly every project and product they've got.

[stephanh]

Re: Doesn't surprise me

This is in fact a normal way of working for Perforce. Let me explain.

Perforce is philosophically derived from the RCS/CVS lineage. RCS operates on single files and just keeps a sequence of revisions to that single file. CVS is kind of a hack around RCS to introduce the weird notions that your project might consist of more than one file and that sometimes you might even want to "branch", i.e. not have a linear progression of versions. However fundamentally things are still tracked at a per-file level.

This makes it fundamentally very easy to check out a subset of the files of your project, since the files are in some sense all quite independent. So you *can* have a single repository which contains all different products of your company.

The reason you will *want* to is because $DEITY help you if you ever want to share some code between projects which are NOT using the same repository. Probably best to merge the projects into a single repository then, so better start with a single repository in the first place.

So not only is Perforce a centralised version control system in that it encourages a single repository per project, it encourages in fact A Single Repository *period*.

In contrast, Git tracks changes on a complete repository basis and doesn't deal with changes to individual files. While doing a partial checkout ("sparse checkout") is possible in Git, it is really not recommended. So Git strongly "encourages" you to split your repository in more manageable parts.

[James 47]

The first SCM I learned to use was Perforce. True, there's a steep learning curve but once I got used to it I quite liked it. This was about 5 years ago. Now we use git, and there's also a steep learning curve with that. Sure the mundane checkout, merge, push, pull are straightforward enough, but once you have to do anything out of the ordinary it becomes a similar ^^mindfuck

[leon clarke]

I really hate GIT

But it's better than all the alternatives.

Once you've got your head round it, it does the right job and does it very well, everything integrates with it, and everyone either understands it or knows that they have to put the effort in to understand it as it's an essential check-box on their CV.

But learning it is a complete git. And I can't help thinking that it would be possible to invent a distributed VCS that did what people want but isn't quite so confusing. But now, if they did then no-one would use it as a critical mass of people understand git.

[Anonymous Coward]

Re: I really hate GIT

Can't say I've tried Mercurial, which would be one of the main competitors to git… but I remember git from when it was officially the back-end "plumbing" to be used with a "porcelain" package like cogito. It was officially the "stupid" version control system, hence its name.

It has matured a lot. A big part I suspect in its adoption would be its status as the standard Linux kernel version control system, Linus having started both Linux and git. Prior to git, well for a while some used Bitkeeper, Linux/MIPS used CVS, and much was tracked by hand!

At the time I knew the basics of CVS, and knew Subversion reasonably well. Git took a bit of getting used to being distributed versus centralised, but little by little I began to get to understand it. It is now my go-to version control system.

There's a lot there and rarely does one need to understand it all at once. It isn't without its frustrations, but it's one version control system that wields a lot of power, if one takes the time to get to know it. Bisect and rebase are two really handy tools I find… the former for finding where something broke, the latter for keeping one's local branch in sync with some up-stream branch.

The only distributed system I have used other than Git is Bazaar, and frankly, I find it aptly-named, never did quite understand what was going on there, sort of like Subversion, but distributed.

There are a few others like Darcs and monotone, but my advice to new starters would be to look at Git and Mercurial as those seem to be the two main contenders.

[Anonymous Coward]

Re: I really hate GIT

Change management will never be simple, GIT is initially hard work to get your head around, because it doesn't enforce a specific workflow upon you. It's upto you to come up with what works for your specific organization.

That said, there is GitFlow, which is a very good starting point for most people.

Documentation at Atlassian is as always superb: https://www.atlassian.com/git/tutorials/comparing-workflows/

[Anonymous Coward]

Re: I really hate GIT

That's one of the things I find interesting about this article. Splunk have clearly adopted a variant of GitFlow, with a PR-driven workflow. There's been a slow but steady movement away from this workflow in most places and projects I've seen. To generate a PR you've either got a proliferation of cloned repositories or a proliferation of branches each of which contain code that may or may not pass review.

It's more and more common to find teams sticking another layer in between the source-of-truth repo and the working branches in the form of Gerrit or similar. You end up with what amounts to a patch-driven workflow, and a clear delineation between code in a known-good state and code that is in a review state.

[Bill S. Preston]

Re: I really hate GIT

I see what you did there! It is a real git. But, I have found this tutorial with Sourcetree as the client makes it way easier to learn.

https://www.atlassian.com/git/tutorials/

[cantankerous swineherd]

Re: I really hate GIT

gitless have a good critique of git and may one day be a viable alternative.

[FatGerman]

Re: I really hate GIT

In a former life I made a living automating merging and testing of changes in a company that relied on P4. I, having no previous experience of version control systems, found it very easy to work with and pretty much self-explanatory. I've since tried to use git and simply cannot get my head around it. Nothing I used to find obvious with p4 seems even possible with git.

All that said, I have it on good authority that everything I made a living writing millions of lines of Perl to accomplish, comes as standard with git. But I wonder if any of that is worth the pain of learning it in the first place. I've gone back to good old SVN.

[Anonymous Coward]

Biggest gripe regarding Bitbucket

… it's insistence on "feedback" at times when I'm focussed on getting a job done.

I'll be navigating between different parts of a source tree in the code viewer, highlighting sections of code to reference as hyperlinks in a Jira task comment explaining some piece of code or whatever, then next thing a pop-up appears asking for a rating.

Bugger off!

My mental stack is rather full at those times and the last thing I need is a distracting pop-up trying to overflow it!

[Ian Thomas]

Re: Biggest gripe regarding Bitbucket

When I get that sort of survey, I always give the lowest rating, and sometimes add a bitch about being interrupted.

[Anonymous Coward]

Of course it's not just Perforce

Using Bitbucket/JIRA here, and corporate is trying to push TFS on us. It's laughably bad in comparison.

Yes, we can use Git back-end with TFS, but corporate have already decided to stick with the old Sourcesafe back-end, and whichever you use, it's still really poor compared to the gold standard that is Bitbucket Server.

Sadly corporate don't understand how picking the RIGHT development tools can mean the make of break of a project. Price and corporate policies dictate what we end up with, not what makes sense.

[Richard 12]

Re: Of course it's not just Perforce

Sourcesafe?

Do they hate your code and just want it to die? We've had several occasions where the database got corrupted somehow and had to be rolled back to an earlier state.

The second time I had a local git-based backup (intended for branching and merging) so pushed up the lost changes again.

It never seemed all that stable either, though that was probably really caused by poor Internet connectivity.

[Anonymous Coward]

Re: Of course it's not just Perforce

Sad to say, the default VCS backend for TFS is Sourcesafe :-(. It might not say it on the box, but if you use it, it's very clear that's it's heritage, it's got all the same horrible limitations.

To compound TFS problems, the web front-end has to work with Microsoft crap browser, so no nice dynamic web app stuff going on like BitBucket has.

Stick TFS is a really poor cousin to BitBucket Server, even if you use Git backend. It's defect tracking is dire too, and CI build system, even V-Next is a poor man's Bamboo. It also costs 10x the price...

I wouldn't be surprised to see Microsoft acquire Atlassian, their product is vastly superior, and they are still affordable, and git seems to be a common thread in all the chatter within TFS circles at Redmond.

[richardcox13]

Re: Of course it's not just Perforce

> TFS, but corporate have already decided to stick with the old Sourcesafe back-end

TFS's non-git backend bares as much relationship to SourceSafe as CVS does.

None of the limitations apply, none of the ongoing issues apply (no need for a weekly analyse to fix corruption, etc).

[Anonymous Coward]

Re: Of course it's not just Perforce

Sorry, but TFSVC backend is Sourcesafe, it's the exact same workflow model, where you have to explicitly check out files before the modify them, where it leaves things read only and all hell breaks loose in third party tools as a result. Infact everything about how it operates is identical to sourcesafe. So it's either IS sourcesafe, or Microsoft have written from scratch something that has all the same limitations as Sourcesafe....

They may have fixed the corrupted data issues, but operationally it's still horrible, and it's like working CVS but with more limitations. I forced to use TFS (with TFSVC aka SoirceSafe) on occasion, and it not possible to do real work there, just single file, no parallel development check out, modify, check in workflow, anything else, the tool is a nightmare.

[phunksta]

Smerforce

I was a big fan of Perforce up to a point.

One day we found it had mixed up a code unit, leaving the wrong version in a build although the tooling told us it was up to date.

After a long battle and a good kicking from maangement we finally found the problem and had to manually patch the file into place as the tools refused to acknolwedge that the wrong version was there.

I could have forgiven this, but I contacted their support and sales (as they were a 'free small team' user but may have adopted the tools company wide). After an initial warm reception from sales, we got support who essentially 'gave up' after 1 email back and forth. Perhaps the direct evidence contained that their tools were unreliable caused some sort of internal conflict? Whatever, the support guy 'went on holiday' and the case went cold. Even after pressing the sales director, we were roundly stonewalled.

One of the main reasons comapnies use proprietary software vendors is to ensure when things go wrong that there is someone to ring and get help from. Unfortunately in my experience, when things go wrong, Perforce just go quiet.

[Richard 12]

Re: Smerforce

I kept on pressing them.

For line endings, it took a week to get them to actually consider the problem at all, and about another month before they admitted that there was a serious bug - but even then they said the bug was the documentation and that they way it actually working how it should. Which is a stupid way that no real user would ever want...

For the other big issues, about four months of back-and-forth before they finally suggested an undocumented hidden function - simply to list the commit text.

At that point we gave up. It still doesn't quite work, and it was clear by then that Perforce just cannot tell you your history.

[jillesvangurp]

Git is sort of a baselevel of sanity these days. It's not perfect but it sort of integrates with all important build tools, everybody can work with it, and it does what it needs to do. There are a few similar version management tools out there that you might legitimately argue are good alternatives but stuff like svn, perforce, and cvs are not what I would consider a sane or even defendable choices at this point.

It's very simple: either you care about version management and you use something decent or you don't and you might as well use something decent. There are very few technical reasons to stick with legacy version management technology beyond "we can't be bothered to give a shit".

Usually when you encounter those, the organizations that use them tend to have loads of other issues that all revolve around having a lot of technical debt; usually fueled by years of mismanagement.

Not the most attractive work place in other words.

[thomn8r]

Obligatory: https://imgs.xkcd.com/comics/git.png

[Kevin McMurtrie]

Six of one, half a dozen of the other

I've used both P4 and GIT and my preference is to Perforce. Git does everything, absolutely everything, including things that should never be done. Very common activities become a struggle with Git's cryptic commands and terminology that usually end up like the XKCD comic says. There's nothing like having Git put thousands of conflict markers in your new code because it didn't quite get the right options. Perforce makes easy things easy so screwing it up takes a bit more effort.

The other issue is shared libraries. Perforce can push and pull a consistent snapshot of many repositories at once. Git can't, so it's pure hell when multiple people are refactoring code across multiple repositories at once.

I've seen nasty bugs in both so I have no preference on stability.

[Anonymous Coward]

Re: Six of one, half a dozen of the other

use a decent GIT GUI, and most things go away, Sourcetree or GitKraken.

[p4jbw]

A Brief Response from the Referenced Author

First, let me say I’m pleased to see Helix covered by The Register, as it provides an opportunity to clear up some misconceptions about the product. In this brief response, I’d like to call out some factual inaccuracies and explain how the main objection is no longer valid.

Regarding factual inaccuracies, Helix is neither strictly centralized nor proprietary. It offers the most advanced DVCS capabilities available, including features Git and other systems don’t (and may never) have. Pieces of Helix are already open source, and I expect more will be over time.

Further, Helix integrates with a large number of the best-of-breed tools across the ALM chain, offers multiple APIs to craft your own, and performs at scale like no other system. Helix supports tens of thousands of users, hundreds of millions of daily transactions, billions of files, and can manage petabytes of storage. I’ve neither seen nor even heard of projects using Git operate at that kind of scale, because it simply doesn’t perform. This is why our customers daily continue to choose Helix over Git.

But as to main thrust of the article, it is fair to say that Helix did not have a good GitHub-style workflow two years ago. But that’s precisely why Perforce has subsequently released GitSwarm, a tool that offers exactly the kind of workflow touted in the article. As such, the article's main criticism is no longer valid.

Finally, Perforce is under new management and takes Git very seriously. In fact, we’re reworking all our Git tooling while continuing to push the boundaries of DVCS science. Stay tuned for more exciting stuff. I’ll later publish a longer response on the Perforce blog on this article for those who want more details.

[diodesign]

Re: A Brief Response from the Referenced Author

Full disclosure: p4jbw is a perforce.com employee.

Thing is, Splunk was not using Perforce Helix, they were using pre-Helix Perforce, but you're talking about Helix, so.... ?

Also, you say your software is not "strictly centralized" but your own publicity reads: "You are probably quite familiar with Perforce’s centralized architecture." So... ?

And you say your software is not "strictly proprietary" but parts of it are closed source. So... ?

Normally the words "factually inaccurate" are the sole factual inaccuracies. Thanks for showing our readers the kind of tedious nitpicking-to-death journalists go through on a daily basis with vendors.

Thanks for calling.

C.

[Anonymous Coward]

Re: A Brief Response from the Referenced Author

He/she also convinniently ignores Git-LFS extensions.....

[Richard 12]

Re: A Brief Response from the Referenced Author

Thanks.

When will Perforce properly deal with line endings?

When will it be capable of showing the labels in the history of a file or workspace?

- Or even handling large numbers of labels without falling over. "Lightweight" changelist labels are a start, but I can't see those in the GUI at all from a workspace!

When will it be capable of showing the *full* commit history of the *project* - see the "railtrack" git demo screenshots for what I mean.

When will it be able to show the history of stream "imports" in any way, shape or form?

- It does seem to be tracked but I have no idea what point that actually gets stored, and neither do your support.

These are necessary tools for multi-developer, multi-platform projects, that are either horribly naive or sadly lacking in Perforce.

That's before considering refactoring when files are split up. Git tracks that - while Perforce barely manages a rename!

Perforce/Helix is radically behind - DVCS is not just about local versioning.

- Unfortunately yes, I am rather bitter. Perforce/Helix has burned out at least one of our good guys, and cost me personally over a hundred hours of work in a few months, fixing depots and working around its shortcomings. We finally do have a workflow that mostly works, but it works despite the tool, not because of it.

[Anonymous Coward]

I guess we will see fewer Splunk releases and many more bugs - which is sad, since it is fundamentally a really good product.

Atlassian reminds me of the Sirius Cybernetic Corporation - in particular the Nutri-Matic drink dispenser. What is does is almost, but not quite, entirely unlike what is asked for.