How the UK's national memory lives in a ROBOT in Kew

The UK’s National Archives in Kew have enough gems and hidden secrets to keep Indiana Jones or Robert Langdon in sequels for the next couple of centuries, with everything from the Domesday book to the UK’s official UFO records socked away safely in its sanctum. But for the swashbuckling archaeological hero of the future, whips …

[Steve Foster]

Oi!

"Sometimes it’s rhubarb crumble."

I'd far rather have that than the abomination that is bread-and-butter pudding (with or without added chocolate).

[Anonymous Coward]

Re: Oi!

Oi!

Bread-and-butter pudding is nice. With raisins.

[Steve Foster]

@skelband

Why on earth would you waste perfectly good raisins on bread-and-butter pudding? Much better to use them in something worth having, like eccles cakes or a proper tea loaf.

[phuzz]

Re: Oi!

I spy another el Reg comparison/flame war coming on.

What's *your* favourite pudding?

(My mum's blackberry and apple pudding is clearly best of all)

[Martin Budden]

Re: Oi!

Depending on the season, my mum's blackberry and apple crumble or my mum's Chrishtmash* pudding.

*this is the correct spelling due to the van-load of brandy somehow crammed into one pudding.

[Steve Foster]

@Martin Budden

Of course, the minimum ratio for the brandy is 5:1. That's 5 parts to the chef, and one to the pudding. Any less and you're definitely doing it wrong.

(icon demonstrating what happens when the lit match is applied at the table)

[Steve Foster]

@phuzz

In general, pretty much any pudding made by someone else for you is "the best" (bread-and-butter pudding being the notable exception of course).

And for anyone else contemplating stepping into this one, desserts are right out.

[Christian Berger]

Re: Oi!

Actually now knowing either of them, both sound delicious.

[Benjol]

Re: Oi!

Quite!

How is having rhubarb crumble not being lucky?

With ice cream

And custard too.

[Graham 24]

Lots of dead trees

Unfortunately, given how hardware and software changes over the years, it seems to me the only real way to make sure all this stuff is archived for future generations is to print it. We can read the Domesday book from a thousand years ago, because it's on paper*. We're having trouble reading the documents from the Olympics 16 months ago because they're in an electronic format that we weren't prepared for.

* or sheep skin or whatever it was back then.

[Primus Secundus Tertius]

Re: Lots of dead trees

Yes, but all that parchment was fearfully expensive. So you had acts of vandalism such as the works of Archimedes being erased and replaced with minor religious texts.

Centuries later, similar things happened with BBC videotapes of Hancock's Half Hour and Dr Who. The Sage of East Cheam turned into an un-person!

"Intelligent computers" (if that is not an oxymoron) should be able to sort out Word 2, Word Perfect 5, etc. Latex can be more difficult if there are dozens of 'include' files which have been 'lost'. Spreadsheets and databases may take a little longer. Does anyone really want those Powerpoint nonsenses? (Yes, if you are a serious archivist.) Then there are Abiword and Gnumeric, Amstrad Locoscript, ... And there is an awful lot of image-mode PDF lurking about.

[alain williams]

Re: Lots of dead trees

given how hardware and software changes over the years, it seems to me the only real way to make sure all this stuff is archived for future generations is to print it.

That is why the UK government should madate the use of a properly specified document format such as .odt (Open Document Format). The use of proprietary or poorly documented formats such as that various changing Microsoft .doc/.docx should be outlawed (they are not properly documented in spite of Microsoft committing fraud by buying enough ISO voting countries to be awarded is ISO number).

The same should be applied to other storage mechanisms, eg: CAD & GIS - proprietary formats abound and the major vendors frequently change them so that users constantly need to buy new versions to be able to read what people send them.

[maconrad]

Re: Lots of dead trees

How do you print a CAD model, database, website, etc?

[hammarbtyp]

Slightly off topic

But I attended a talk by the company responsible for running http://www.legislation.gov.uk/.

It was fascinating description of the problems of converting century year old documents into an XML frormat and then using that to create a sophisticated searchable database. Also they have to turn around new legislation quite quickly.

The result is a great public resource containing all present UK legislation going back to the earliest one still on record(1267 since you ask).

It also goes to show that not all government IT projects are disasters waiting to happen

[Peter Simpson 1]

Re: Slightly off topic

The trick to avoiding disaster is to hire a firm that knows something about successfully completing large IT projects, as opposed to a firm (e.g.: CGI Federal) whose area of expertise is limited to winning government contracts.

[Dave Harvey]

Re: Slightly off topic

As I regularly say when talking about the disaster which was NPfIT, and in particular the London contract - "that well-known healthcare company called BT"

[durandal]

Re: Slightly off topic

I enjoy the fact that Hansard has been archived on the slightly scary sounding 'millbanksystems.com', like some sort of black operation based out of the bowels of Thames House.

(http://hansard.millbanksystems.com/)

[Jemma]

i wonder..

Since its already been mentioned that for various reasons long term storage on digital media is going to be an issue, not least because of media degradation maybe we could go a slightly different route.

Since modern papers aren't much better than tapes & the like why not use metal storage? Something like a stainless steel cartridge vetsion of the old punch tapes or Jacquard cards & a high speed reader, after all most just sits there for years doing nowt.

The problem here is mirrored all over the world at the moment in all fields. Will there be anything left of 1990-onwards in 800 years? probably not, we are, bar buildings maybe producing so little that's tangible and what we do produce generally is designed to fail.. including the tape drives and other devices Kew needs to use..

We might be in a latter historians eye - the mysterious denizens of Dark Ages II: The Search for a .DOCX converter... kinda sad if so..

[JDX]

Re: i wonder..

You can read most of the content in a DOCX file with notepad if you wish to.

[Anonymous Coward]

Re: i wonder..

"You can read most of the content in a DOCX file with notepad if you wish to."

Maybe so if you have a pre-historic binary computer to hand. Here in the 69th century we don't, we do conveniently have a time machine, but it's use is restricted to making frivolous comments on El'Reg, not historical research.

[JDX]

Re: i wonder..

So how is that better or worse than writing it down and finding that in 7000AD we don't use writing anymore?

[rh587]

Re: i wonder..

"Since modern papers aren't much better than tapes & the like why not use metal storage?"

Paper is fine. If it's acid-free archive stock, not the bleached rubbish most people feed through their printers.

Plenty of paper records have survived through history because they were made of substantial paperstock (they couldn't make it any thinner with contemporary technology), and the surviving ones are the examples here paper makers struck on a pH neutral/slightly alkaline formula, giving the paper a stable chemistry (whether they realised it or not at the time). All covered by ISO9706. One manufacturer claims a 200year guarantee (provided it's stored properly, not in the bathroom), which I think shows remarkable faith in their business prospects! We can rest easy knowing our descendants will be able to sue their descendants if our archive crumbles to dust in a mere 195 years...

That said, a ream of that stuff costs £20+, so a substantial markup on normal 80gsm office fodder. You wouldn't want to go printing the Internet on it...

[Tom_]

We should be archiving it on the moon, so it survives any Earth based disaster that's small enough to leave the moon's surface unharmed. eg. climate change, many levels of asteroid impact, supervolcano, gamma ray burst, etc. Even if all life on Earth is wiped out, the sum of our knowledge will remain available for any life that comes along later, either from elsewhere or starting from scratch locally.

I propose a system that starts with extremely large scale symbols, visible to the naked eye from the surface of the Earth - enough to inspire curiosity, so as to encourage further investigation. Then smaller symbols, still large enough to be seen with primitive optics, which explain some basic science. As concepts get more advanced, the symbols can be smaller, as they'll have already explained how to develop better telescopes. Smaller symbols can be duplicated over more of the moon's surface to allow for redundancy. At the stage where rocketry, orbital mechanics, etc. are sufficiently explained, everything else can be stored in a bunch of duplicated vaults - readily available for direct physical examination.

Large symbols can be written with nuclear weapons and smaller ones with orbital lasers and rovers.

/bosh

[Uncle Slacky]

Or we could embed it in a feckoff big black monolith and bury it on the Moon...

[monkeyfish]

And what language do you propose we write it in? Hieroglyphs?

[TheOtherHobbes]

COBOL.

Seems to be almost immortal so far.

[Anonymous Coward]

Massive symbols

"And what language do you propose we write it in? Hieroglyphs?"

A big fuck-off barcode, which will require the construction of a big fuck-off barcode reader, which will emit a big fuck-off beep when future generations scan the moon.

And then a really big fuck-off "unexpected item in bagging area".

[Peter Simpson 1]

Nice touch

Providing recommendations for the thirsty explorer at the end of the article!

Dad spent some time at the old PRO in the 1950s, researching his PhD on the Naval Career of Admiral Sir George Pocock. Which is quite a good read, actually.

[Sandpit]

Dark?

I often think of Dark Archive as oxymoronic. With petabytes going in (somepoint in the future, what is it now?) and nothing coming out do we really have an archive?

Any digital archive must assume that the media WILL fail, simply because, well, it will.

[Anonymous Coward]

Google would love to do it for us!

[Mike Brown]

and we should probably let them.

[Anonymous Coward]

They probably slap 'copyright Google' over all the digitised data and then charge us a fortune to access their copyright data.

[John Brown (no body)]

"They probably slap 'copyright Google' over all the digitised data and then charge us a fortune to access their copyright data."

Scarily, that would almost certainly be legal for them to do without a carefully worded contract. The originals would still be available to personal visitors, but Googles derivative work (scanned + maybe processed) would be their own copyright.

[monkeyfish]

Furture of web archiving?

"This is the future of web archiving," says Thomas, picking up a tape, adding, "at the moment".

So that would be the, er, present of web archiving?

[Yet Another Anonymous coward]

Computers in the basement

Sir Humphrey: .... a few others lost in the flood of 1967.

[to Humphrey] Was 1967 a particularly bad winter?

Sir Humphrey: No a marvellous winter, we lost no end of embarrassing files.

[imanidiot]

Other POI nearby

Kew Bridge Steam Museum.Just a bit away on the other side of the Thames. For those mechanically inclined and interested in steam power it is highly recommended.

http://www.kbsm.org/

[Sir Sham Cad]

Informative article.

The bit about the pubs. I shall have to seek these out in the spring.

[The Wegie]

Re: Informative article.

The Botanist is also a microbrewery with a range of Kew-themed ales. Additionally, what was The Rose and Crown on the other side of the Green now does an excellent range of craft beers.

[Christian Berger]

Of course the Iluminati archive is located in Bavaria

I mean why should it be in the UK. It's in a Panthenon shaped building in the Upper Paliatine.

[Primus Secundus Tertius]

Re: Of course the Iluminati archive is located in Bavaria

Or encrypted in the DNA of an Upper Cambrian trilobite.

[Jes.e]

What protections..

What protections are there against demonically possessed Word macro viruses in that Dark Store?

[WraithCadmus]

Re: What protections..

HMG has The Laundry to deal with these aspects of government projects.

[OzBob]

This sort of reminds me of concorde

ie. the British simply beavering away on a simple concept that is complex to implement, but doing it quite well and without too much self-congratulatory fanfare.

Excellent Article, A+

[Dave Bell]

I recall obtaining what was left of my grandfather's military record from the National Archive. Much had gone up in flames in the Blitz, but there was still his medal card. giving his service number, and that let me find when his Military Medal was gazetted.

It's all very well to talk about fire protection and floods, and there's the implication that data on computer systems can be backed up, but, while we don't need to worry about the Luftwaffe any more, does it really make sense for the Dark Archive to be the only copy of the data? Maybe we don't need to know where the back-ups are, but I hope there is an off-site back-up.

[JDC]

Another pub

The Coach and Horses on the Green, nice in winter with its open fire.

[David Thomas]

Thanks for the article and comments. Just a couple of responses to comments. TNA has offsite back up somewhere in England. I'm not sure printing to paper is helpful - TNA has almost as many pages of digital data as it does physical so a huge storage problem. Also many IT formats such as websites are hard to print in a meaningful way. As for metal storage, TNA has some 1940s audio recordings on metal tape. Can we read them? No we can't.

And pubs - try the Express just across Kew Bridge.

David Thomas

[ElNumbre]

Microfilm

A warehouse full of microfilm, plus some instructions on building a reader seems to my little head to be the best 'backup' strategy of still being able to read a document even if 'digital' file formats change. Ultimately if future generations can manage to build a microscope/magnifying glass and a light source, the data should be reconstructible.

[Yet Another Anonymous coward]

Re: Microfilm

Balanced against that is the chance of anyone bothering to read it - so just as effectively "lost"

Having information like the census data online means it can be data mined to research patterns of immigration, social mobility etc etc. Having it on 1000s of microfilms in different places is more secure but means the data will almost certainly never be accessed or used.

[alain williams]

Backup copies ?

I hope that all of those many TB of data are mirrored in a couple of protected data centers a looong way away from London. Then nightly rsync to keep them up to date. Probably not very expensive considering what these copies are protecting.

[Frankee Llonnygog]

At the risk of sounding a tad political

Wonderful article about a national institution we can be proud of - an example of what the civil service can do when Ministers don't interfere. I wish it happened more often. Sadly, inevitably, some ministerial berk will set targets for making all parchments digital by default, scanning them at a rate of 500 pages a day, and then shredding all the 'hard copy'.

[Anonymous Coward]

Do more reading!

Suffice to say, it is most likely that all of the server infrastructure is easily replaceable these days and just as importantly the software configurations of said servers and repeatable, no one roles out serious amounts of infrastructure without configuration management tools these days.

The article mentions a little device called a StorageTek SL3000, a little reading would tell you that this device enables them to write to many diverse types of tape media, and have many copies on each type of media. It also allows you to export tapes, which can then be sent off site for later retrieval. Not to mention that it has facilities to constantly check and evaluate the use and wear of the tapes, drives and robots, with intelligent migration. So I would expect that TNA are making use of such facilities to make sure their archive is available for the foreseeable future under any circumstance.

Regards putting things onto Microfilm, that is ludicrous as it often suffers from the same or worse degradation problems than paper, several large archives have announced projects to migrate microfilm to digitised records.

[maconrad]

First National Archives Catalog online.

"The catalogue itself began to move online in 1998, the first national archive in the world to do so."

The United States National Archives and Records Administration had begun to move its catalog online well before 1998. See this entry from the Internet Archive from June 1997:

https://web.archive.org/web/19970606073326/http://www.nara.gov/nara/nail.html