Even Google struggles to balance fast-but-pricey flash and cheap-but-slow hard disks

Google has revealed that it still relies on hard disk drives for most of its storage needs, but has been able to ‘dramatically’ improve the performance of its storage systems with a homebrew automated data tiering system. The ads and search giant admitted its ongoing fondness for spinning rust in a Thursday post that explains …

[Pascal Monett]

“What’s new with Google Cloud’s Storage”

We've found a new way to extract even more information from you with the excuse of improving your storage throughput.

This new API is going to boost your storage efficieny while sending us the filenames and filesizes of your files during downtimes.

[mdubash]

Hierarchical storage

Didn't we call this hierarchical storage--back in the 90s?

[gwp3]

Re: Hierarchical storage

Back in 60's!

[UnknownUnknown]

Re: Hierarchical storage

It’s hardly surprising when mechanical drives are still significantly cheaper with a sweet spot around 16Tb where you can get some excellent Toshiba Enterprise grade 3.5 SATA drives for GBP250 delivered, or cheaper.

[An_Old_Dog]

The New Mechanical Turk

OPR MSG: MOUNT TAPE VSN 168844 ON DRIVE 0C4

It's "near-line" storage.

[Eecahmap]

Re: The New Mechanical Turk

That takes me back. Nearly 40 years, but it takes me back.

[DogRuff]

Re: The New Mechanical Turk

It has been, hasn't it? Pretty much the same for me .. 40 odd years ago .. *>deity<*! Gettin old...

[Luiz Abdala]

Isn't that the point of ZFS?

I'm gonna admit complete ignorance here but... wasn't ZFS designed for this?

(If there is someone that can actually reach a Zettabyte of data to manage, it's Google.)

Another thing: can't they just let an AI figure the tiers out? Enough data for training, they sure do have. If the usage patterns don't help an exact algorithm to manage the tiers, perhaps an AI can do that?

I get it, these guys are handling something that would boggle my mind, but still...

I am confused and humbled by what I don't understand, honestly.

[v13]

Re: Isn't that the point of ZFS?

Not exactly. Colossus is a disturbed storage system that scales without limit. ZFS isn't distributed. You can find an overview here: https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system

[jabuzz]

Re: Isn't that the point of ZFS?

Hate to blow your bubble but ZFS is a toy file system and always has been. To address why Google don't use it, it has no concept of tiering which when operating at the scale that Google does is incredibly important. Secondly it is a single computer only file system. If I can't upgrade all the components of my storage with zero seconds of downtime then it is not an enterprise system. This sort of high availability can only be achieved with clustered or distributed filesystems something ZFS is not and never will be.

[Raymondgu]

Additionally, wouldn't it be easier to have an AI determine the tiers? They definitely have enough data for training. Could artificial intelligence be able to handle tier management if consumption patterns aren't helpful for an exact algorithm?