back to article Wanna keep your data for 1,000 YEARS? No? Hard luck, HDS wants you to anyway

HDS' federal subsidiary says it has a Blu-ray optical storage platform for long term data preservation, with 1,000-year M-DISCs in prospect. The kit - we say kit although no products have been announced yet, just this "platform" - is for US federal agencies who need to "preserve and archive mission-critical data indefinitely …

  1. Pen-y-gors

    1000 years?

    Yep, long term preservation of storage is critical, but I'm not sure if a DVD-like thing that lasts 1000 years is actually going to be much use beyond the lifetime of the Third Reich, that other well-known 1000 year thingy.

    The disk may well survive. Will the equipment to read it still be available? I think not. Will the algorithms to decode it still be known? Remember the Domesday Project.

    And why does the Naval Research Dept need to preserve its data for 1000 years?

    Really, if you want serious long term preservation, print it out on vellum, or chisel it into the stones at Stonehenge! Sad but true...

    1. DropBear
      Trollface

      Re: 1000 years?

      Yeah well the Egyptians tried that, and it sure went well... we'd still be gawking at those hieroglyphs if it weren't for those meddling kids the Rosetta stone...

      1. Vector
        Holmes

        Re: 1000 years?

        "we'd still be gawking at those hieroglyphs..."

        I've often wondered what archeologists several thousand years from now would make of all these shiny silver disks that started to appear in the late twentieth century.

        1. Anonymous Coward
          Anonymous Coward

          Re: 1000 years?

          I'll bet some of you think "cloud" is a real media.

          1. Anonymous Coward
            Anonymous Coward

            Re: 1000 years?

            "I'll bet some of you think "cloud" is a real media."

            The salesman told me it was clouds all the way down!

      2. Captain DaFt

        Re: 1000 years?

        "Yeah well the Egyptians tried that, and it sure went well... we'd still be gawking at those hieroglyphs if it weren't for those meddling kids the Rosetta stone..."

        Considering how determined their conquerors were to eliminate all that 'Heathen' knowledge after ancient Egypt finally fell, I'd say they did a pretty good job of preservation.

        No matter how well you future proof your data, if someone is determined to destroy it, you're going to suffer losses.

        1. Techno Musing

          Re: 1000 years?

          Lots of questions... Why do you migrate data in the first place? Tape-to-tape, disk-to-disk...

          USUALLY! Because one technology or format has become obsolete OR something is wearing out/degrading. The format that comes from the original Compact Disc (CD) is still supported and working today in these new devices. That's over 30 years. What digital technology is still backwards compatible to 1982 (beside a digital clock), both in data & media format, and you can readily/easily buy today?

          If you didn't have to migrate your data because you can still easily read it, use it, have it supported with no special device or service, didn't take up power or cooling, why would you migrate? This is the media I'm talking about.

          You might still buy systems and drives for speed, density, supporting new formats (with support of old formats), etc.

          1. Stoneshop

            Re: 1000 years?

            If you didn't have to migrate your data because you can still easily read it, use it, have it supported with no special device or service, didn't take up power or cooling, why would you migrate? This is the media I'm talking about.

            You forgot one thing: physical storage. Every storage location can only hold so much stuff (and stuff with data on it tends to require qualified storage locations), and if you're running low on space it may well be worth it to migrate if every box of storage media now can hold four times as much data as before.

        2. Anonymous Coward
          Anonymous Coward

          Re: 1000 years?

          Not just others... but proceeding kings in Egypt were keen to destroy past knowledge that made them look bad and others look good, even if it was true and helpful. Remind you of anyone else? :(

          1. Anonymous Coward
            Anonymous Coward

            Re: 1000 years?

            And then there was that ugly little business with Amenhotep IV (aka Akhenaten). After he died, it wasn't his successor but the people at large who poured on the Brain Bleach. It was rather fortunate enough remained to figure out enough of his story to make sense.

    2. Knieriemen

      Re: 1000 years?

      The thing to keep in mind is that with Blu Ray, you won't have to do the periodic (and expensive depending on size) tape migrations.

      Storagebod wrote a great post on the time it takes to do petabyte-scale tape migrations - http://www.theregister.co.uk/2013/03/27/storagebod_petascale_archives/

      1. Yet Another Anonymous coward Silver badge

        Re: 1000 years?

        Avoiding the need to keep up with technology changes by using an optical disk standard is the reason we backup all our data to Laservision

      2. Charles 9

        Re: 1000 years?

        "The thing to keep in mind is that with Blu Ray, you won't have to do the periodic (and expensive depending on size) tape migrations."

        Then what happens when you have an optical disc migration instead? DVD migrated to BluRay, and for archival we'll probably be moving from BluRay to Archival Disc unless something else comes along, and even within Archival Disc there will be several iterations for starters. The vaunted 1TB/side won't be available for a few years yet. Heck, even external hard drive tech like RDX requires periodic migration (RDX claims a 30 year life right now, but can you really believe that number?).

      3. Anonymous Coward
        Anonymous Coward

        Re: 1000 years?

        Please READ that article by StorageBod. He does know what he is talking about.

        He uses the time for TWO drives to migrate the data, one from the other.

        A couple of years ago I managed a real life data migration project, or a petabyte of data, from an older StorageTek library to a new library. Something like a dozen LT03 drives for reading the old data, and four LT05 drives on the writing end. The whole thing took a few days, and went flawlessly.

        You could expect to perform something like that seamlessly if you were simply updating the drives in an existing library.

        1. Stoneshop

          Re: 1000 years?

          Please READ that article by StorageBod. He does know what he is talking about

          Err, migrating a petabyte worth of tape data using only two single drives? I assume that's just for illustration porpoises, otherwise it'd just be daft. Which would then also hold for your statement that he knows what he's talking about, unless you left out a negative there.

          If you have a tape collection that size (1000+ LTO3 cartridges of 3.58 million cat pics each) you don't have just a single tape drive; the very minimum is a two-drive multicartridge loader per technology. The problem that someone is blocked from retrieving data from a tape you're not currently handling also disappears as soon as you have multiple drives, although it will temporarily slow down the migration process.

    3. Primus Secundus Tertius

      Re: 1000 years?

      As for "thousand year thingies", there are several western examples such as England, Iceland, Switzerland. In the orient there are China and Japan.

      Apols to any I have omitted.

    4. Annihilator
      Joke

      Re: 1000 years?

      " Remember the Domesday Project?"

      Nope

      "Really, if you want serious long term preservation, print it out on vellum, or chisel it into the stones at Stonehenge! Sad but true.."

      Or, write down the retreival algorithms and store it with the disc. Probably best to store it on one of the 1000 years discs so it doesn't become corrupt. Thus solving the problem forever.

    5. Techno Musing

      Re: 1000 years?

      Funny you use this analogy. M-Disc is "rock" based material.

      1. illiad

        Re: 1000 years?

        rock??? nah, do the research... 'vitreous carbon' - think carbon nano fibre

        'stone' has a problem, if you bend it, it breaks...

  2. Cipher
    Big Brother

    Lois Lerner...

    ...and ten others at the IRS need to hear about this. Oh, wait, too late right...

  3. Lionel Baden

    It would be nice if humanity could start collecting all its data for preservation.

    next step is to make it available within context to all.

    im thinking from a future standpoint not current.

    it would be an awesome project to be a part of though !!!

  4. tirk
    Coat

    In 3014, future historians open the archive to discover...

    Rick Astley?

    1. ItsNotMe
      Facepalm

      Re: In 3014, future historians open the archive to discover...

      Rick Astley?

      Or Farcebook files...or even worse...all of the narcissistic tripe from Twatter? Oh goody.

  5. Brian Miller

    M-Disk: 42,000 pictures of cats

    From their own website: 21 hours of non-HD video, 120 minutes of HD video, or 42K pictures of your cats.

    So in 1,000 years, the archivists will pop one of these into a drive, and see pictures of cats. And they will wonder what the hell is wrong with us!

    1. DropBear

      Re: M-Disk: 42,000 pictures of cats

      ...their boffins will simply conclude cats were sacred animals in our times.

      1. Pascal Monett Silver badge
        Coat

        What, they're not ?

        1. ItsNotMe
          Devil

          Re: What, they're not ?

          Sure they are. Everyone loves a little Pussy...don't they?

      2. Annihilator

        Re: M-Disk: 42,000 pictures of cats

        Maybe they weren't so sacred in the ancient Egyptian days either then?

    2. Stoneshop

      Re: M-Disk: 42,000 pictures of cats

      or 42K pictures of your cats.

      Elsewhere on their site it says:

      Using estimates with average file sizes, each M-DISC™ DVD:

      Holds up to 8,000 photos

      Holds up to 1,200 songs

      Holds up to 240 minutes of video

      Holds up to 100,000 documents

      So, it apparently takes just over 5 cat pictures to comprise a photo

  6. Salamander

    So in a thousand years, the US Navy will still want to look at combat data from today. That is like the British army still wanting to analyse combat data from the Battle of Hastings.

    Why is it that some people have such amazing optimism?

    1. Charles 9

      Hey, people still look up Sun Tzu, don't they? Historical combat data can have its uses in the broader scheme of things.

  7. cosymart
    Facepalm

    1,000 years

    Load of bollocks. How do you prove it? Just look at the technology and sociological advances made in the last 1,000 years. Look at the equivalent from c1080. The Doomsday Book, this was written in Medieval Latin and was highly abbreviated and included some vernacular native terms without Latin equivalents. So, it was written in ink on velum? and we still cannot read/understand all of it.

    How many words and abbreviations in regular use in various Navies (vernacular native terms seems quite apt :-) ) will be understood in 100 years time let let alone 1,000 years.

    Anyone paying good money for 1,000 year archives needs to invest in quality paper, ink and people who have legible handwriting! Oh, and add a dictionary to the list of things to be transcribed :-)

    1. Techno Musing

      Re: 1,000 years

      If the bits (or pages) are gone, you can't even argue your point. You won't even get to prove it's garbage.

  8. Crisp

    The discs last 1000 years

    But how long do the reader writer machines last for?

    1. Paw Bokenfohr

      Re: The discs last 1000 years

      Doesn't matter with M-DISC, as they can be read in a standard DVD drive, only the writer has to be special.

      In 2047, when the last company making NX-Ray (TM) drives announces they're no longer going to make them (they had backwards compatibility to DVD, BluRay and UV-Ray (TM) you see) then whoever still has a (useless) archive on M-DISC will just read it off and transfer to Data Crystal storage.

    2. Primus Secundus Tertius

      Re: The discs last 1000 years

      Of course a DVD reader built today will not be around in a workable state after 1000 years. But the disk could be analysed and a new reader built.

      Some time back I read an article about an optical reader that could follow the track on an old (pre-vinyl) gramophone record and reproduce the music minus most of the background noise.

    3. Techno Musing

      Re: The discs last 1000 years

      It doesn't matter! With removable media, you can throw the drive away, buy a shiny new one, and still read your data. This has been working with CDs since 1982.

  9. RaidOne

    If these BD discs are like DVD-Rs that I used in the early '00s

    They will last maybe 5 years, then all files will be corrupted.That is a problem with organic media. Pressed DVDs are a lot better, but it seems that the aluminum inside corrodes as well.

    HDDs are not much better, hysteresis will kill the differences between 0s and 1s, but much slower. All my backups moved from DVDs to HDDs.

    I am not sure about SSDs, maybe someone can illuminate me on that. Not that I can afford to backup on SSDs yet.

    1. Flocke Kroes Silver badge

      SSD

      Without power, an SSD will eventually collect more bit flips than the error correction can recover. With power, the firmware can read through all sectors, and re-write the ones that were hard to read. Eventually that re-writing will wear out all the reserve capacity and the device will lose data. The manufacturers can plot a graph of life time verses temperature for high temperatures and extrapolate a guess at the life time for sensible temperatures. Real figures take years to collect, and by that time people have moved on to the next generation of technology.

      Ages ago I read about an archive project that wrote new firmware for their disk drives. The firmware automatically distributed copies of the data across the array and monitored the health of the other drives. As long as people swap worn out drives for new ones, the data should stay in the archive, and upgrading to newer technology can be done one drive at a time. The most obvious points of failure are lack of funding and someone accidently deleting everything. I still have more confidence in that lasting 50 years than 1000 year optical disks.

      1. Anonymous Coward
        Anonymous Coward

        Re: SSD

        "Ages ago I read about an archive project that wrote new firmware for their disk drives. The firmware automatically distributed copies of the data across the array and monitored the health of the other drives. As long as people swap worn out drives for new ones, the data should stay in the archive, and upgrading to newer technology can be done one drive at a time. The most obvious points of failure are lack of funding and someone accidently deleting everything. I still have more confidence in that lasting 50 years than 1000 year optical disks."

        One issue is when new technology comes out, like faster SAS speeds or an entire replacement. Then you just cannot swap drives out but a full-on migration.

        1. Charles 9

          Re: SSD

          "One issue is when new technology comes out, like faster SAS speeds or an entire replacement. Then you just cannot swap drives out but a full-on migration."

          But you can still perform it gradually. The big part is replacing the controller tech with one that can bridge the gap, say one with the new tech built in and the old stuff supported with a module. Then you can change out to the new drives as you swap out the old ones. Once the last old drive is gone, the module can go as well.

    2. Techno Musing

      Re: If these BD discs are like DVD-Rs that I used in the early '00s

      Don't compare "consumer" grade media to archive/enterprise grade media. Huge difference in materials used and longevity.

  10. btrower

    Take a hint from nature

    You are all of you huge backups of a system, parts of which are millions of years old. The tried and true method used to preserve the shark design these past few hundred million years is how our older data is essentially already being preserved as we copy data we care about from one generation of technology to the next.

    Could we design a living thing to carry copies of data of interest into the future? Probably. Meantime, though, we need to ensure that as time marches on we make copies and copies and copies.

    I have stuff on tape and disks from decades ago. It is basically not recoverable and some stuff was actually lost. Going with a specific media is not the answer. The answer is to have an ongoing program of backing up and copying forward and to do that you need, not rarefied exotic storage meant to last a thousand years, but rather a ubiquitous interface connected to the rest of the world. I vote for the Internet. That's where real men put their backups, after all.

    Stuff written in the Reg may last longer on paper if someone bothers to preserve them. However, they would have to bother. This way, with it sitting on their disks, being archived at archive.org and no doubt gems like my comments being copied off by others for posterity and scholarly research, it is both preserved and actually available to use. The reason? It is being copied forward to ever cheaper storage because it is easy to copy a rack of hard-disks compared to preserving stacks of boxes.

    1. Charles 9

      Re: Take a hint from nature

      As I recall, DNA is an inexact kind of thing. Which is why even identical twins don't have identical fingerprints. In any event, while sharks may not have evolved much over a few hundred million years, we'd probably be able to note some incremental steps along the way, meaning the copying process isn't very exact.

      1. phil dude
        Boffin

        Re: Take a hint from nature

        Yes, DNA copying is deliberately a little bit noisy - E.coil 1 error every 10e-5 ,but Humans have the same error rate but better error correction. By now the analogy with tape comes up! How many bits error correction do you need?

        The physics of the universe is a very good way to store information - in other words you can't beat entropy, and chemistry is entropy patterns encoded in electron orbitals.

        A self-replicating knowledge machine would have to be thermodynamically stable in (or around) the final state of your information. That, essentially, is the problem that biology solves. We are all thermodynamically probable molecular machines - made out of atoms that simply have no choice but be atoms.

        Your eye, for example, doesn't have that choice as it needs all the bits! Many of them get replaced every day...think about that when you wake and open your eyes - no "booting now" screen!!

        Petroglyph is the best form of storage, probably a QR code. Just look at some of the 300 million yr old fossils that are still living with the same body plan...

        P.

        1. Techno Musing

          Re: Take a hint from nature

          The last article I read on DNA storage, it cost $35K/MB to write.

          1. Anonymous Coward
            Joke

            Re: Take a hint from nature

            "The last article I read on DNA storage, it cost $35K/MB to write."

            I can store that for you at $35K per year... as long as it's to be stored in human form...

  11. AcrossThePond

    Compatibility across media

    While 1000 years is ridiculous, the idea of storing on media designed to last beyond 50 years makes total sense. BluRay and M-Disc are compatible in the same drives for both read and write. The biggest difference is M-Disc is about half a generation behind standard BluRay discs from a size and cost perspective.

    Beyond 50 years is not out of the question though. In the same BluRay drive that supports M-Disc and 100GB+ BDXL discs, you can still insert and play your very first commercially produced CD. http://abbaoncd.wordpress.com/european-cds/european-cds-1982/

    1. harmjschoonhoven
      Coat

      Re: "you can still insert and play your very first commercially produced CD"

      I remember a story form New Scientist that the very first CDs could not survive transport in the cargobay of an airplane, because the lower airpressure drew bubbles in the CD. Of course the problem was solved in subsequent pressings of CDs.

      My coat is the one with the Hollerith cards in the pocket.

      1. harmjschoonhoven

        Re: Hollerith cards in the pocket

        Just for the sake of the argument, 1 TByte of data can be stored in a block of 30 m x 30 m x 30 m of punched cards. Mothballs not included.

        1. Stoneshop

          Re: Hollerith cards in the pocket

          Paper tape can hold ten bytes per square inch, which at 0.1mm thickness would result in a data density of 100 bytes per 64.5 mm^3, or roughly 1478MB per m^3. So that's 709 m^3 for a terabyte, a cube a little under 9m each side.

          And it's silverfish you need to guard against.

    2. Techno Musing

      Re: Compatibility across media

      Milleniata announced last week that M-Disc technology is now at 3 layer BDXL 100GB and is 100% compatible with standard drives. Though most drives today can now write the older DVD M-Disc. With Blu-ray, there is now distinction.

      Everything centers and gravitates to the most common high volume capabilities.

  12. chivo243 Silver badge
    Joke

    Only if I were...

    Daneel Olivaw, then this kind of storage make sense, think of all the series over that time span you'd need stored.

  13. Techno Musing

    Stuff that no longer exists

    Did you know that the Oil & Gas industry (and other survey agencies an preservation societal organizations) has warehouses full of media (tapes, film, paper, disks, discs, etc.) with data on it of places and things that no longer exist.

    Imagine, pictures, surveys, seismic data, sensor data, and so forth, of places, objects, sites and views that will never exist again. This data has been deemed a national treasure by the US Government and needs to be preserved forever, or as long as possible.

    How many of you know what Mt. Rushmore looked like before it became a monument?

  14. Anonymous Coward
    Big Brother

    Why go to all that trouble!!

    Just throw around a few mentions of jihad and Al Qaeda and a fictitious friendship with Angela Merkel and the NSA will hang onto your data in perpetuity!!

    Your tax Dollars at work!!!

    1. Vector
      Black Helicopters

      Re: Why go to all that trouble!!

      Yeah, but they might supply room and board as well.

  15. Filippo Silver badge

    1000 years for the media, how much for the reader?

    It's kinda pointless to have a support that lasts 1000 years. You can reasonably prove that the support will last 1000 years, if stored properly.

    But you certainly can't prove that the *readers* will still be around that far in the future; arguably, that's extremely unlikely. At some point, the economics of building them will fail. The cost to keep replacing them will climb steeply. Eventually, it will become cheaper to just copy all the data to a new "1000 years tech" - which will succumb to the same fate way before 1000 years have passed. And so on and so forth.

    Basically, there's no way you can avoid having to migrate all the data every now and then. You can try to reduce the number of times you need to do so, but the reliability of the support is not the major factor for this (not beyond a few decades). The fact that these disks can be read using standard readers is probably much more useful than their 1000 years duration. But, arguably, you could use BRs, get pretty much the same result, and save a load of money.

  16. 420Penguin

    Unreadable File Formats

    OK, assume the data survives 1000 years. What file formats are they stored in? Something where there was a documented and agreed upon International standard... or some mysterious, undocumented, proprietary file format by a company long out of business as will be the case with Microsoft Office docs? Try opening a Word doc in 1000 years! (or 20!). Where possible, data should be preserved in open standard formats like ODF or PDF.

    1. Techno Musing

      Re: Unreadable File Formats

      ...or store the documents and/or other content in multiple formats. That's what preservationists do.

  17. dieseltaylor

    M-disk user

    I have just given copies of my sons wedding photos on an M-disk to my son. To my daughter the wedding and her 2002 holiday travelling around South America. I think everyone who cares about important information should be putting it on the most robust media - not forgetting that the media should be in a format that is widely readable.

    Given the writers are quite cheap it seems daft not to use an archive quality disk for important data.

  18. GrumpyOldMan

    Long term archive concers

    Historians are very concerned about the long-term fate of things like digital photographs and there is real concern that thousands of images of every day life will be lost to future generations - although how many selfies, babies, and cats as well as thousands of really bad wanabe stars on old YouTube vids future generations will want to look at I have no idea! Everyone has them stored across multiple devices, I have approaching 200,000 digital images as a long-term photographer - but how many do I actually look at? I started in digital in 2002, and am fanatical about organising my images by date. But I still have thousands of printed photos and negatives properly stored in archive mounts in boxes of albums going back to the early-mid 70's - my teens. They are still as good as the day they were printed, no messing about with moving them onto the latest format, no worries about reading the algorithm - just hold the negs up to the light. We have access to photos taken hundreds of years ago and need only eyes to read them. I'm now looking at getting mine transfered to photobooks but the cost is pretty high as is storing them!

    I now use dng workflow for my digital photos, as like LTO it (supposedly) guarantees long-term backward compatibility for RAW files and is an open format, but how long is long term? And to the guy who said 'put the algorithm to read the disk on the disk..." er.... duh! So this 1000 year thing sounds great - until someone in 5 years time comes up with the next greatest 10,000 year archive, and so on.

    The whole topic of language evolution is also an interesting one. Language has changed a great deal in the last 1000 years, and is changing even faster as communication gets faster. My 16 year-old daughter comes out with new phrases almost daily now (and an awful lot of rubbish). Even her 'cool' older sister at 21 feels old and out of touch!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon