Facebook's request to the flash industry: 'Make the worst flash possible' Flash-memory designers may currently be focused solely on improving speed and endurance, but Facebook's director of capacity engineering and analysis wants something completely different from them. "The Facebook ask of the industry is make the worst flash possible," Facebook's Jason Taylor told his keynote audience at the …
"but I don't dare throw away, because I might need that data someday."
No need to worry, 1980's porn has evolved, today we have moving pictures........
PS : I have upgraded, replaced all of my drives, work and home, with SSDs. ( Well except for the the USB archive/backup attached to the NAS). I don't even remember the sound of the click, click, scratch, screech as data is being trawled from the anals of the highly defragged IDE 30 Mb "whopper" of a drive......
"At least the women back then looked like women."
No they all looked like failed Cybil Sheppards, too much make-up, big hair and half the vaseline budget went on covering the camera lens for soft-focus shots!
Now they don't even bother with all that guff, drug up a couple of 15 year-old runaways on Sunset Strip, get them to "act" in a hastily rented house, set up a Nikon D3/Canon 5D3 on a stready-cam shoulder mount and start shooting HD porn! (*)
(*) Note that I'm not an expert on shooting low-budget adult fare, regardless of the investigation I've done when I'm bored!
Many years ago, I went through all my old hard drives, user-created CDs and DVDs, SD cards, and USB flash drives, and basically just dumped their contents onto my (at the time) extremely large 1TB hard drive and then erased them and donated them to charity or threw them out, as appropriate.
After the most rudimentary de-duplication by hand (i.e., only copy my user folder from old hard drives, and try not to end up with 18 copies of my entire MP3 collection), the entire unorganized collection ended up being less than 200GB. The process required an adapter to mount a bare hard drive via USB ($20) and maybe 2 afternoons at most. I made a big directory called "backups" with a bunch of subdirectories called e.g. "White SanDisk 2008" and then just copied away.
The benefits are obviously less clutter, plus now I only have one main drive with all my data to secure/back up, plus I can search for documents from 15+ years ago in seconds, plus I'm not worried that if somebody breaks into my home they might find a loose unencrypted drive with valuable personal information. Highly recommended.
Because, just like magnetic media slowly loses it's magnetic polarisation due to the earths magnetic field,
flash-memory loses it's electrical charge over time.
Okay... it may take 10 to 20 years but still... the data may need to be read and re-written every 5 years or so...
You can say that about pretty much all user generated content on the web - Facebook just happens to be storing by far the most of it this write-only shite.
I'd describe the whole of Facebook itself as "very sad", but then I never saw the point of it when it first appeared, and still don't today.
"I'd describe the whole of Facebook itself as "very sad", but then I never saw the point of it when it first appeared, and still don't today."
Well aren't you the cool one for avoiding it. Here, let me get you started:
- No longer a need to stand and pose while 17 cameras are used so everyone gets a picture of an event
- No need to hope your mates will share pictures of events
- One update and everyone knows the plan, no need to call around, text around etc.
- If plans change everyone sees immediately and nobody is left out of the loop (except you, you're not on FB)
- I don't need to travel 10,000 miles a week or spend hundreds on phone bills to keep in touch with friends
- My photo album is enormous, and I know who is in the pics because they are all tagged, and I can contact them immediately when reminiscing over the photos
- After Uni everyone went separate ways and contact is easily lost. Facebook means I always have an up to date phone directory and email address
- FB groups have replaced Yahoo groups for communities and clubs. It's amazing the people you meet online when you share interests
- Sharing an interesting link with friends
I'm sure there are more, that was just off the top of my head a few things to help you see the point. Of course you're one of the cool kids so will now explain how you can do all of that but using email and ICQ chat in twice the time with half the friends...
Also worth noting that a friend of mine back home often gets left out of things because someone needs to remember to call or text him - everyone else just sees the broadcast and comes to the pub.
@Lusty - I'm old, very old - possibly enough to be your granddad but me and my mates use FB in that way - and its a damn sight easier to get every one to arrive at a damp field somewhere in Europe (either to climb a lump of rock or look at each other's cars) than any other method.
Sorry if I have now made it uncool for you as well.
Fashionable to hate Facebook, isn't it.
If you've tried it, genuinely don't like it, then fine, I respect that. But, there are about a billion of us (I guess you'd call us idiots) who do have a use for it.
If, however, you're just an old miser who likes to moan about everything, then fuck off back to the good old days of the 19th century, where women and the poor knew their place, England ruled the world, and the only content which was "user generated" was quite literally shit.
(Twitter, on the other hand, even I don't get...)
If you've tried it, genuinely don't like it, then fine
Yes I tried it, recently in fact (earlier this year) as I recalled a lot of friends that were raving about it way back in 2010, so I looked up their entries and guess what - not one of them has posted a single update since about 2011/2012.
I looked up about two dozen people and there's just tumble weed rolling through their pages.
I'm sure Facebook has its uses, and appeals to certain groups, but like the guy from Facebook said - "The majority of that data will probably be written once and read never" - which chimes with my personal experience, nobody (I know) is using it any more. They've all moved on, Facebook for them was just a fad.
(Twitter, on the other hand, even I don't get...)
I actually find Twitter quite useful, following people (not necessarily friends) that have interesting opinions or commentary. Although anyone posting more than half a dozen tweets a day will usually get the chop in no short order.
The Director of Capacity Engineering at the largest social site on the planet, a company which not just begs to have people constantly upload stuff but actually does everything it can to tie everything about you in a single location has just said that most of it is crap that no one will *ever* see or care about.
The takeaway from this for FB users should be "No one cares. Really, they don't."
Some content is temporary in nature.
E.g. two weeks ago I posted a picture of a burger with a note that says "and this joint have wi-fi". I knew that a friend of mine would be travelling the same way a week later and my picture gave him a feel for what kind of food to expect and what other services were available. And sure enough, a week later he made a stop at that location. I had also tested a shortcut which I told him about and he subsequently followed.
So a week ago my silly burger picture had value, but now it has become largely irrelevant and a year from now possibly just garbage. For all I know, the burger joint that opened in 2012 could be closed by the end of this year or they could radically change their menu or discontinue their offer of free wi-fi. The shortcut that was good this year might be terrible road work infested territory next summer. (in that area of the country they don't mess about -- they completely remove the asphalt leaving only large pebbles around that threaten to punch holes through your car)
In short: What has value today might be worthless next month. Someone might find my hamburger post (w/comments) worth a peek five years down the road, but I agree it is doubtful. Maybe I'll drive through there again some day and look at my posts from around that time to refresh my memory? Either way, why not store it in a sensible fashion?
This post has been deleted by its author
Then load into scanners when you need to read it ?
This would also require that more tree's be planted to provide the pulp required to produce the storage medium (paper), consuming C02 in the process.
By contrast, flash memory consumes irreplaceable raw materials and only ever contributes CO2 during manufacture and subsequent use.
The ask of FB is stupid and no one calls them out?!? Instead people only sat they want it?!?
This is what tape was invented for! With LTO now at 2.5 TB (yes TB) per cartridge (without compression), over 6TB with, no other form of long term storage data is going to come close to this any time soon. Yes tape is slow ... so what. FB itself says that people don't use it much. Tape systems (supported by hierarchical storage) are the exact solution to this. Density, cost and energy use (none while the tape sits there waiting for some to pull up grumpy cat).
This industry just disappoints me more and more every day.
Speed is the key for this use case scenario. Whilst tape is fine for bulk storage, its useless for giving up its data in a timely fashion. I'd imagine most FB users would want to only wait ~30 seconds to get that cat picture back. If FB adopted an 'archive to tape' model, perhaps along the lines of Amazon AWS Glacier where it can take 'some' hours to retrieve data, then firstly people would forget they'd requested the data, and secondly get slated for 'losing' information.
For personal data backup, Amazon S3 then Glacier is perfect for me. Because it takes time to upload and then costs a nominal monthly cost to store, it forces me to be disciplined to only keep anything of value.
Maybe the numbers don't support this, but I would think that from a power usage point of view, tape would be more costly. While it's not being used, flash can use no power as well but, unlike tape, power usage to retrieve something when needed is very low.
Also, you'd need to manage the indexing and robotic tape libraries, etc... which adds complexity, if nothing else. I know that at volume that's pretty cheap but I get the impression that they are using 'conventional' disks and are okay with the price, they just want to see if flash can approach the price of normal drives if requirements for speed and read/write cycles are relaxed.
The expectation is that at some point this data could be read as otherwise why keep it at all? Very little of this data WILL be read again but all of it MIGHT be read so if it is they still need to maintain some kind of reasonable latency. I can't see tapes really meeting that.
Presumably they know their own requirements so it seems fair enough to me.
Essentially, they are asking the question: How cheap can you make flash if we don't care about performance or re-write cycles?
I would go with a slightly different approach for a drive that would suit my needs perfectly...
small section of high grade flash + massive section of prom (programmable read only memory)
have the drive make any initial writes to flash and allow it to stay there for a period of time (30 days), anything that is unchanged over those 30 days move into prom (Write Once Read Many)
this moving of data would be handled internally by the drive speed of transfer from flash to prom would not be an issue since the user would be unaware of the change and read access would be similar speeds regardless of whether the drive was pulling from prom or flash.
It seems to me that the controllers for such a system already exist in the current generation of hybrid drives where commonly used files are held in flash with the remainder held on spinning platters, my suggestion would replace those platters with prom
They'll love to have it.
Think of all those pictures of kitties and other inane objects they will have to wade through just in case one of them might have a child in it. Then they can go after you for distributing child porn. In second thoughts perhaps not but it would be a good way to DDOS the spooks, flood them with data. (Just an idea and in no way am I suggesting that it might possibly, sometime in the future happen ok)
And essentially dedup.
Gradually remove all those different pictures of beaches, or a kitten attacking a ball of wool, and replace them with just one iconic image!
Eventually you will just have three pictures: a kitten, a dog and a naked lady on a disk somewhere.
"The Facebook ask of the industry is make the worst flash possible," Facebook's Jason Taylor told his keynote audience at the Flash Memory Summit on Tuesday in Santa Clara, California. "Just make it dense and cheap."
Uncoincidentally, "dense and cheap" is a perfect description of the typical Facebook user.
Let's discuss the flash solution. It could be made in small or large modules, each would have its own advantages and drawbacks. Even though it's frugal, flash still needs power to function. The larger the basic module, the more power it will use. Then there's the matter of reliability. Larger modules would fail more frequently per module than small ones. And ultimately, they would be more expensive per byte because they would need more complex controllers. These factors would favor smaller modules. However, smaller modules would require more complex routing, switching, finally it would need very complex controllers per each brick of modules.
Would it be cheaper than current flash technologies? Sure. Would it be cheap? Not by a long stretch. Flash is still 8-10 times more expensive than spinning drives. TLC doesn't bring the cost down far enough.
It's also not a matter of density. At the same node, I suppose flash makers could make features denser, but even if they were twice as dense (which is rather unrealistic), we're looking at only four times the raw capacity -- which is still more expensive than spinning media.
Interference would become a greater problem, and it would probably cause the usable capacity to not increase as fast as raw capacity did. Durability would suffer, of course, but as the guy said, it's not a problem for them, especially since they already don't delete the content, but keep it hidden.
Nevertheless, it's still not a solution. Perhaps Facebook will be happy with the resultant module, even if it's expensive, if they think it will save power, or if it would be less complex to build and maintain, but I don't think so.
--
Which brings me to tape. There are T10000C drives that offer 5 TB per tape, and T10000D on the horizon which will offer more -- that's beyond the LTO roadmap at the moment, so I'm not talking about LTO. Tape has the nice property that when it's idle, it's not using up power and when a cartridge is needed, automation takes care of picking it up and mounting on a drive.
--
That said, I realize that if he said that waiting times for spinning up disks are too long, waiting half a minute or so to access a tape would probably be much too long for a user to wait. Caching part of the content on disk to wait until a tape is mounted would probably alleviate some of this concern. However, the service is free of charge, so Facebook pretty much has all power to set SLAs for it.
Read-from-deep-archive times are the key here; what if your kitty pic is in the middle of a tape that's in a rack 10m from the drive? User Gets Bored, "FB sux".
Flash doesn't need power when it's not being written or read either - I think dynamic power management would be front and centre of the storage device design.
What I can't get my head round, though, is the implied contract of FB, Google/YouTube, Flickr an' all with their users is to keep all their data, however much they like, forever. Can they keep that up indefinitely? If not, who is going to prune it?
So let me get this straight. Facebook has a perfectly good solution right now (racks of dormant hard drives, using essentially no power) but they want to improve access latency.
Your suggestion is to switch to tape which will increase latency by an order of magnitude.
Did I miss something?
@AC 17:54
No mate, you got it in one.
Facebook are happy with the cost and power consumption of their existing solution and just want to find out if they can, for a similar price, use 'better' drives.
Replacing large banks of magnetic hard drives with SSDs is presumably a SIGNIFICANTLY simpler task than replacing them with a tape-based solution because all the logic is unchanged - just swap the hardware. None of their processes would be affected at all.
If they get what they are asking for, all that happens is they replace like-for-like hardware and almost instantly get better I/O, without having to implement another storage tier or additional caching; just new drives and faster access.
A sensible retention schedule might shift their paradigm. No access requests for 5 years, bin it. If it was that important, the pea-brained user will have uploaded it from their own device, which they can then backup to their own choice of media, if they can avoid either breathing or chewing gum for long enough. The same issue affects every organisation that stores useless data, but doesn't have a sensible policy or strategy for binning it.
4gb of 200kb photos (fb limit) is about 20,000 kittypix. Are people so technically inept that they can't burn a dvd of their timeless masterpieces to show to their fellow kittypix anoraks, in much the same way that holiday slides were employed to lethal effect in the olden times.. Who needs a will to live anyway?
SD usually cards seem to use crappy and cheap flash memory. Get whoever's making these chips to put them in an SSD for you, Facebook.
I just checked on Newegg and the cheapest $/GB I could find for SD cards was 53 cents/GB, compared to 80 cents/GB for an SSD.
Then again, hard drives are 5 cents/GB.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
