Re: Eh?
You are quite right, you can't rebuild a 4TB single disk in minutes. It's utterly impossible. However, what you can do is have a distributed system such that 4TB of "lost" data is actually lots of much smaller segments. Let's say, 100 x 40GB each, each of which is part of a redundant set of some sort. Then you re-assemble those 100 x 40GB segments on 100 different disks using spare capacity. It's certainly possible (in theory) to write 40GB to a single disk in less than 10 minutes.
That's the basic principle, you use some form of distributed redundancy system such that if you ever lose one disk, then you can invoke 1 very large number in both the recreation of the "lost" data and in the reconstruction.
Of course you can do this with a clever file system, but it's also certainly possible with block-mode arrays too. File systems can be rather more clever than that as it's only the active data that needs to be recovered. A native block mode device, at best, can only know what blocks have been written to via a bit map. However, doing all this requires a lot of heavy processing and data shuffling. You can't simply delegate the task to some dedicated hardware RAID controller to do in the background.
That 4TB recovered in 10 minutes still involves (at least) reading and writing a total of 8TB of data (depending on how the redundancy is generated), and that's at least 13GB per second. With more complex, and space-efficient redundancy schemes, you may have to read several times that to recreate the data, so it would be even higher. That's very demanding, and will require a lot of CPU and memory bandwidth which also, of course, has to carry the normal workload at the same time. Of course, if you relax that 10 mins restore time, it's less demanding.
It was my experience of storage arrays (mid and enterprise) that the processing capacity was often the limiting factor on I/O performance (depending on access patterns).
Biting the hand that feeds IT
