All very lovely...
But does it work? Can, say, a large University or a national Revenue service put their trust in HPE's 3PARs?
HPE says its latest 3PAR OS, v 3.3.1, has better data reduction, faster iSCI networking, upgraded data protection and an extra helping of automation to help admin staff. The Adaptive Data Reduction (ADR) feature can reduce capacity needs by up to 75 per cent*, it claims. It includes both in-line deduplication and compression, …
that the KCL 3PARs are taking part in some sort of very distributed SAN, using SANITY software. A project that was underway before the problems. I've no idea if this is any good or what the technical jubbery-doings are of this. Any comments from anywhere would be welcome. Sounds like a good thing, TBH.
From an insider accounts of the events and my personal experience at KCL a couple of years ago I can tell you that there was no fancy distributed arrays or something involved - simple case of one 3PAR array corrupting the data after a crash during a controller node replacement.
Looks like the cause with the Australia tax revenue office (or whatever it's named) was similar.
It's seems hard for HPE to do the right thing. The extra CRCs might prevent data corruption that was being missed. Bless them for trying. Fingers crossed everyone upgrades and it works before more suffer data lose. All inclusive licenses seems like a good idea - why did no one ever do that before, oh, they did, years ago, at least HPE is trying to play catchup.
So who else is providing end (host) to end (disk) data integrity in hardware ?
Certainly none of the startups and even the established vendors usually only support a subset of this and only in their highest end platforms. 3PAR has had this internally for years now using T10-DIF and added T10-PI with the 8000 and 20000 series, this is just an enhancement to that end to end data protection that pretty much everyone else is lacking.
If it's so good how come there are multiple data lose cases in the public domain? Why introduce extra CRCs if they aren't trying to fix something that is broken? Seems rude to make sweeping statements about what the other players can't match if you can't manage rule number 1 of storage - don't lose the data...
Take a look at T10-PI it's about guaranteeing end to end data integrity beyond the array, but hardware can and does fail that's why people have DR plans. The larger the install base and the bigger the customer the more corner cases you'll encounter, there are also many external environmental and people & process factors that can undermine pretty much all of the above.
Step back a few months and a leading vendor was doing data destructive upgrades, a couple of years further and they had very public data loss events on both of their leading platforms.
I'm not sure how pretending T10 PI is unimportant because others don't have it or that it somehow relates to failures you have no real knowledge of is particularly constructive. You should really be pushing your preferred vendor to support the same, or maybe data integrity just isn't that important to you as claimed.
I've not suggested CRCs (T10 PI or others) are unimportant, of course they are and lots of vendors have used T10 PI for a long time, Dell, IBM, NetApp, Fujitsu to name a few. IT10 PI is not special to 3PAR and is not a silver bullet, the whole picture has to be considered. Data protection requires a host of good measures and it the the whole that needs considering. For example NetApp have just introduced triple parity, Isilon can protect against up to four drive failures in the same data strip and object stores like CleverSafe claim protection for up to six failures. Ex-NetApp Recovery Monkey wrote an interesting piece on it last year. http://recoverymonkey.org/2016/12/01/uncompromising-resiliency/ The 'none of startups' you condemn on mass may want to pipe up to correct your insult - I suspect a few of them do take data protection more seriously than you suggest.
Everyone has CRC's in one form or other, neither CRC's or T10-Dif are equivalent to T10 PI as they don't provide end to end verification only point to point.
Netapp & IBM seem to have been grandfathered support from LSI Engenio on Eseries and low end DS only, Dell on PERC controllers, so we're not exactly talking ubiquitous mainstream enterprise storage support and I'm not entirely sure I condemned or insulted any startups I simply stated a fact.
As for the silver bullet I didn't claim it was, it's merely an enhancement to what was already there. If anything you suggested 3PAR required this to fix an availability hole, whereas T10 PI has always been there along will all the other availability options.
e.g. multi parity raid, enclosure level availability, memory fencing, persistent cache, persistent ports, replication, storage metro clustering etc which I believe is part of that whole picture you suggested needs to be considered.
This post has been deleted by its author
So, I am surprised there wasnt more noise/details about that? I expected HP to dance all over the bar once that announced?
That 75% thing being tied to the "get thin" thing makes me wonder if this really the efficiency we've been looking for though? ;)
Happy to be proven wrong, so please share details.
Daniel - NetApp
What you mean like the poorly advertised fact that until very recently Netapp's inline dedupe only actually deduped the contents of cache ? Even now only dedupes cache and most recently used data requiring traditional post process dedupe to achieve anywhere near that of any other inline system.
It's the old "up to" bollocks. Like "up to" 30 MBit broadband when you'll only get 2. Or "up to" 90% off when 99% of things on sale have 10% off.
Why don't they just say "we use LZ4 just like everyone else so you'll get much the same compression ratio"?
The devil is ultimately in the detail. Compression rates depend on the dictionary size. Dedup rates also do, but both forms of data reduction depend entirely on the methodology used. What impact they have on performance is yet another matter.
Seems like HPE aren't shouting about these things but are instead trying to quantify administrative effort. I wonder why.
Compression rates depend on the dictionary size. Dedup rates also do, but both forms of data reduction depend entirely on the methodology used.
I'd suggest the type of data has just as big or bigger impact.
E.G. raw (not compressed using a lossy format like h.264/5) terabyte or greater video from a movie studio I'd suggest would have bad lossless compression and little dedupe capability (unless multiple identical copies are kept on the same array).
Or for really bad dedupe (but good compression) what about 400-600 GB/day of application/webserver/firewall/audit logs, where each log is full of unique data (customer reference numbers, receipt numbers, login id's, transaction numbers, date stamps to the millisecond, etc), I'm sure you'd get 75% dedupe on that - not.
Not sure what the confusion is surely 75% savings = 2:1 dedupe + 2:1 compression without the benefits of thin, zero detect etc required. So you could state it as 75% saving or a 4:1 compaction ratio.
Of course YMMV based on data mix but I believe Netapp and many others provide a similar average ratio guarantees, of course all will have their system specific caveats.
Compression would seem to have many more use cases than dedupe alone, making those ratios much more achievable with both less upfront analyses and less risk for both parties.
It's free and non disruptive to adopt so seems like a win win unless you have an axe to grind. Judging by the focus on compression in these comments it looks like this last missing piece has really touched a nerve :-)
Yes they counted thin provisioning in previous guarantees but were always transparent about that.
"The Adaptive Data Reduction (ADR) feature can reduce capacity needs by up to 75 per cent"
So ADR is compression, dedupe and data packing so that's the 4:1, if you then want to add thin provisioning savings obviously the ratio potentially increases as it would do for and is quoted by everyone else. But the metric used will depend on whether you're interpreting this as overall savings or effective capacity.
Whatever way you look at compaction, cost is really the deciding factor and compression just reduced 3PAR's cost significantly which in most cases was already competitive without the need for a 4:1 guarantee.
Your comment doesn't make a lot of sense, both 7000 series and 8000 series use intel CPU's and that's where the compression is taking place, so pointing the finger at the ASIC although fun for the competition requires a bit of a leap of faith:-)
Please show us these many proofs and maybe tell Intel and Google while you're at it because they're all over the ASIC and FPGA market at present.
It's another 60% so once you have both on the array stops, turn zero detect on and things go backwards. :-) Honestly any person who told you that has no business selling or supporting any kind of storage.
Dedupe and compression use CPU, memory and backend IO on any platform, how much will depend on the architecture, mix of data, algorithms in use and a few hundred other factors. If you can't turn it off then you'll never know the impact so you can always claim it's zero but those CPU's and memory are there for a reason. Throw an unfriendly workload at it and watch as dedupe and compression simply burn CPU cycles and eat memory to little effect.
But the simple answer is take it for a test drive and run a POC or even a bake off, you'll be pleasantly surprised how it stacks up vs the competition and that's before you look at the other all inclusive data services.
Inline compression and dedupe do use CPU/memory, nothing is free, but done right they should reduce backend IO. If they are using extra backend IO vs. being turned off then something is very wrong and I'd be surprised if this was the case on 3PAR. On 3PAR dedupe can take 40% of the performance - other platforms may be less, the same or more - nothing is free. The question stands, does anyone know what is the likely performance impact of turning compression on is on a 3PAR?
Yes back end IO can generally be reduced but for dedupe if you have match you have to do an additional IO to verify this isn't a potential hash collision, someone not doing this is playing Russian roulette with your data. Similarly dependent on the implementation garbage collection can create a lot of additional back end workload as can additional post process compaction.
If you have a workload that doesn't suit dedupe or compression and you insist on doing so you're also going to see a lot of overhead to the point you array will likely choke. I did see this recently at a customer with an almost always on array, not really the arrays fault per se the array was undersized and the workload was a poor candidate, but the fact these features couldn't be tweaked or disabled meant moving the workload elsewhere.
Some of the details between platforms will differ in terms of implementation but I would have thought the compression overhead would be pretty similar. LZ4 compression on any box running Intel CPU's with offload extensions should generate a pretty similar load per controller, the multi node architecture and ASIC could maybe help here by offloading the CPU and distributing the load. But from a host perspective compression will be an asynchronous process handled after cache acknowledgement so should be relatively transparent.
If the initial hash is too lightweight lots of candidates will need lots of verification, generating backend IO, an entirely safe initial hash doesn't scale to today's data sets. There is a lot of space between these two for a reasonable initial hash so IMO any dedupe that increases backed IO is a poor design.
Garbage collection is nothing to do with dedupe, but you are correct, done poorly it can create a nightmare of a performance profile.
I entirely agree there are data sets that don't suit dedupe or compression. For these cases it is an advantage if these features can be turned off on a per LUN/volume (whatever the vendor calls it). Having some of these data sets should not not make an array 'choke'. Compression information at least should be available at a granular level to help identify the application and set specific characteristics such as non compression if appropriate. (Bigger point, storage should be application orientated, and provide information on a per application basis as well as LUN/volume.) If an array is undersized it will cause issues - arrays should be able to upgrade performance and capacity without interruption to service. In an ideal world easily from the entry level unit to the most powerful. (Scale out may can help here.)
LZ4 is very common as it is CPU friendly. Other algorithms may compress more but are heavier of CPU. LZ4 is however not CPU free, so there is going to be an overhead on any system of turning on compression unless you have custom ASICs handling it and that is more expensive than a bigger CPU. BTW the Intel chipset can do encryption at little general CPU overhead (a few vendors use this) but it does not have a similar LZ4 offload built in. Given the Gen5 ASIC doesn't do the compression and the Gen6 platform is not yet announced (or confirmed if it will do the compression) the question remains does anyone know the performance impact of turning compression on with 3PAR? (It's not 60% or nothing.)
I'm not sure you'll get a straight answer because at the end of the day it probably depends on many factors such as workload, data type, platform etc but I'm sure HPE will have numbers available for customers, so maybe speak to your HPE team.
If that's not an option then assuming the other players e.g. EMC, Netapp, Pure, Nimble etc publicly release their own overheads, then you could do some research and probably extrapolate from there.
"Because ASIC development slow down innovation. This has been proven many times over the last 15 years!"
Innovation is great when it makes sense or when it is required. The problem with the storage industry is that vendors are so desperate to be "innovative" that they bring half baked products with bells and whistles to the market. The industry has spend 50+ years innovating ontop of spinning rust. They'll spent the next few years innovating ontop of flash.
For such a stagnant industry the true innovation has come from material sciences (flash) and now the software engineers are trying to make it go a bit faster. But that's not innovation - its an incremental improvement.
The word innovation is used for marketing purposes and has little to do whats going on inside the box.
Most of the customers would prefer stability over "innovation".
And when dedupe and compression costs CPU cycles ? - so what, just throw another CPU at it. Whats a 3000$ CPU in a 100.000$ system? The problem here is that the storage guys havent figured out SMP yet, because they were so busy "innovating".
Biting the hand that feeds IT © 1998–2020