Dropbox slips 500PB into its Magic Pocket, not spread over AWS

Dropbox has sucked the vast majority of its data off Amazon's cloud servers and into its own custom storage centers. This shunting of customers' files has been dubbed Project Magic Pocket. In a blog post on Monday, Akhil Gupta, Dropbox's veep of engineering, said his company has been an early adopter of Amazon S3 to store bulk …

[Electron Shepherd]

Those damm amateur coders...

there's nothing in the open source community that's proven to work reliably at our scale.

Yes, it's simply outrageous that no-one has decided to use their own free time, and coding just for the love of it, to develop something that Dropbox can use without having to pay for it.

[Adam 52]

Re: Those damm amateur coders...

I don't think he was complaining, and yet if he hadn't said it there'd be a whole load of open-source advocates saying "why didn't they use HDFS/MongoDB/whatever... I built my home object store in a weekend it's easy" and complaining about them using proprietary technology.

[Anonymous Coward]

Re: Those damm amateur coders...

Yeah, Dropbox are far too community-minded to release their own code for all those freetards to mess up.

[Kraggy]

Isn't Dropbox the company Snowden warned about? Didn't they take on to their board an ex-politican provably hostile to privacy?

I guess if they have all 'your' data in-house it's easier for their friends at the NSA to get at it.

[Anonymous Coward]

Nothing wrong with Dropbox if you encrypt before you upload to it. Snowden has a point though; SpiderOak also encrypts files stored locally.

[Anonymous Coward]

Oh, I say!

>there's nothing in the open source community that's proven to work reliably<

Fair point, well made.

[batfastad]

Easy

Well it's easy to build an in-house storage infrastructure when you give only 50MB for free.

Mine's the one with an HDD in the pocket, connected to owncloud, rsynced offsite.

[Anonymous Coward]

Re: Easy

I'm quite sure they give away more than 50 MB.

I'm not a freetard. I'd like to pay for their service, and have around 100-200GB synced over several machines. The problem is that their minimum paid plan is too much -- one terabyte.

I've tried Dropbox, OneDrive, Google Drive, iCloud -- OneDrive keeps forgetting some files (and duplicating others), Google Drive asks for my password every few minutes, iCloud does not work across all machines I use. Dropbox works without a problem.

I don't have the resources for a OwnCloud system. I've been using a WD My Cloud sync app, it is working OK so far, but I cannot get it to connect when I'm using some wi-fi networks.

[Netbofia]

Re: Easy

just setup syncthing. Put a pi on you network with a disk or two voila.

[calmeilles]

Re: Easy

A terabyte is too much does strike me as an odd response.

If it's functional and affordable then surely 1 TB at £80/year is a better deal than say Amazon's 500GB for £160/year even if you don't expect to fill it.

[Anonymous Coward]

Re: A terabyte is too much does strike me as an odd response.

If I don't need it I don't want to pay for it.

If they had a plan like 200GB for 1/4 or 1/3 of the price I'd go for it. And I'll be able to sync 200GB in all computers I need.

[Anonymous Coward]

Tech media shocked: There's another way to do cloud?

I'll say it again like a broken record... there is more than one way to cloud. AWS does not define the only way to provide cloud services.

[benderama]

At least with AWS they had multiple layers of security, redundancy, and support. Now that they're on their own iron... Who knows what can go bad?

[John Robson]

Now they do, rather than Amazon being the only ones...

I still don't...

[bdj]

Out of interest, and as I'll let someone else do the googlefu but what impact does this have to AWS and it's offerings? 500PB seems like a awful lot of lost business?

[Doogie Howser MD]

Project Magic Pocket

Now if they'd called it Project Wizard's Sleeve, that would have been cool...

[Paul Smith]

In house

Why is there any surprise that they brought it in house? They require a physical storage for lots of infrequently accessed data. That requires infrastructure and costs no matter who physically controls it so they either pay some one else enough to make a profit doing it or pay for it themselves and keep the profit someone else would make.

[Bob H]

Re: In house

I read a posting by some company who had some hosted hardware in NY, I can't remember who it was but I think I found it through El Reg though.

Interestingly their argument was that if you could justify the headcount to manage hardware and you had predictable capacity then outsourcing your servers doesn't necessarily make sense. Every argument I have seen about going cloud either boils down to two things: scaling/flexibility and/or administration overhead. Whenever I hear the administration overhead argument it is put forward by software people because, understandably, software people don't want to care about hardware.

But simply put, the costs of hardware and hosting can be cheaper than the cloud if you have the right economies of scale. But people should be doing more diligence than just saying "put it all in the cloud!" because they might not be doing the best for their business.

[Rainer]

It should be noted that despite discounts, having 500PB on amazon probably wasn't cheap to begin with.

Also, amazon's infrastructure is on-demand only in theory.

If you potentially need to have lots of machines provisioned, you've got to pay for that privilege.

Same is almost certainly true for storage.

[Phil Kingston]

I've not seen "durability" measured before. I wonder what they're referring to? Anyone care to enlighten me?

[Bob H]

I believe it relates to how much data is "lost" through bit-rot or other data loss factors, rather than just being unavailable.

[mfh]

Yes, Availability vs Durability is the difference between temporary and permanent data loss. Four nines of availability allows for 4.5 minutes of inaccessibility per month (~ 1 hour per annum).

What I find interesting is that their numbers (500PB & 500,000,000 customers) works out to ~ 1TB/customer. Of course this is raw footprint, so there is quite a bit of protection overhead, but still that seems quite high.