Running queries on the HMRC database fiasco

When it comes to talking about last week's data loss by the HMRC, I was told not to use precious words outlining my feelings of rage and bafflement that a government body can be so cavalier with so much data because, presumably, we all feel the same. So I will simply note, for the record, that my gob has been totally smacked …

[Paul]

What it costs vs what it costs

I still think you're being generous at £500 ; this also assumes that HMRC don't employ anyone who is a DBA :)

But I think everyone who posts "how much it would cost" might be missing the point - it's obvious to anyone the cost of doing the actual work is peanuts.

Many of these contracts are scams - see Computer Weekly for endless examples of such. Sometimes costs are loaded post completion to get the headline cost or overrun down (as with Capitas CRB deal where endless repeated CRBs are required).

Companies with financial nous get lock in deals which allow them to later milk the public sector at will. PFIs are another example ; companies charge inflated sums for any work not specifically in the contract.

It strikes me as highly likely that this quoted figure is not the actual cost of doing it, but the cost of getting *EDS* to do it - and the reason it is so high is because EDS can more or less charge what they like, because the contracts state that *only* EDS can do it.

[Anonymous Coward]

Password & encryption

Alistair Darling said on the Today programme that the discs were "password protected but not encrypted". He seemed pretty clear about the distinction.

[Paul]

Post?

Why on earth was it posted?

Are they still living in the dark ages? With all the technology available today there are MANY alternatives.

Surely a VPN connection to a DMZ with a 'secure' access area for sensitive file transfer would have been better. Access details sent via an encrypted email. The technology is so simple that any home user can set it up if they put in a bit of effort. The MOD sites have a secure network, couldn't government departments use this, even if they only get access to a DMZ area tagged on the side? The technology is there but why aren't they using it?

Burning disks just does not make sense.

[Bruce Leyden]

If I had a pound for every time...

Shit! If I got paid that much for chopping up a CSV file...

*goes misty-eyed*

[Ray Foulkes]

HMRC IT is outsourced, the outsourcer has to be profitable!

There is no point in thinking that you could offer do the work for £500 unless you offer to do it for CapGemeni ; see: http://www.channelregister.co.uk/2007/11/13/capgemini_job_cuts/

I think that there might be a markup put on your work as well as some overhead of writing the requirements, formulating the demand for extra IT work when costs are being reduced, 5 layers of management to get the request approved after choosing the appropriate budget line, possibly a committee or two to pass through, then the costed proposal from the outsourcer, the quality plan for the work, fully detailed PERT chart, test specifications, approval process, proposed modifications to service level agreements, approval of tea breaks (oops, only kidding) etc. etc. etc.

Techies like you make things sound EASY. ;-))

[Marty]

So it appears

So it appears that the way the child benefit department has a massive database, and they don’t know how to use it.

They have outsourced the management of the database to a company, which is probably owned by some relative of a government minister.

They and only they know how to query the database, which will keep them in a job for a very long time.

They decide how much to charge for one of their employees to do any sort of work on the database. And because said lazy bastard can’t be bothered to do the work they are asked to do, they quote some ridiculous sum.

When asked how they can do it cheaper, for half the price, they will click on export as CFV. Copy said file on a couple of blank CD’s in a password protected zip file……

Excuse my lack of faith in yet another government department.

I think I'll start keeping my cash under the matress, and when i have enough money I'll be off somewhere safe.....

[Graham]

select count(idiot) from HMRC

I'd suggest that the reason that NAO was being sent the data in this format was that it was a format that HMRC already had and was already using internally in some integration or other. Given that the extract contained *all* the data, I'd suspect that this was a copy of the feed to an HMRC data warehouse.

The most likely formats therefore might be an XML file, a CSV file (either of which really isn't going to trouble a black-hat much to figure out the format, given they've probably saved it with a file extension of .xml or .csv) or a database extract. You have to remember the thinking of the numpties might be, "well here's the data, no we don't know how to get it onto your machine, but it is on the disk; oh you'll incur an expense getting it? Sorry, not our problem". But as you say, let's notr discuss it... pointless anyway seeing as I'm sure we can all guess where the "password" protection comes in.

And the Id cards argument from Darling is just hilarious, in a very black way... yes, it would have made it less likely that the NAO would have had to request the data in the first place (bearing in mind that they didn't actually request the data at all) but on the flip side once the idiot with unrestricted access to the dataset sent the whole caboodle through an insecure channel we'd be in an even worse position than we are now. And it doesn't matter how many technical safeguards you try to put into your system, at some level there will be some idiot with too much access who doesn't understand why what they are about to do should qualify them for a red hot poker up the jacksie.

I'm still trying to figure out who are the bigger set of morons here, the HMRC clowns or the policitcal muppets, it's a difficult choice.

[Anonymous John]

According to yesterday's Telegraph

The NAO (not having a mainframe) then passed the database to KPMG to process.

So the NAO was paying a profit-making business to extract usable data instead of HMRC doing it in-house. A rather convoluted way of auditing a Governemnt department.

Little or no cost saving there.

[James Joy]

Lazyness

Part of the problem is that in my experience the goverment either outsource there IT, or due to rates of pay, are not employing the cream of the IT crowd.

It would appear reading between the lines that either there IT staff did not know how to extract the data, or there out sources over complicated the job.

I have come across a couple of firms that when we have asked them for some data from there system it has been a case of:-

1. We need to apoint a project manager.

2. They will write a specification for the job.

3. You must then approve the specification.

4. A programmer will then develop this.

5. This will be test by there testing team.

6. It is then run on the system so we can get the data.

They say they could not subset the data, however, they refer to the current records. Therefore, how did they remove all the old records?

Assuming that for what ever reason that it is impossible to extract just part of the data, there are several tools on the market that allow you to capture the printable output. This can then be convered to a format where they can remove the data that is not needed.

At the end of the day it is all down to lazyness. If they really wanted to only send them the information that they asked for then they could have done.

Worse case they could have told them to come to there office and make the selection. After all they actually one wanted to audit 0.0004% of the records. Again too lazy to send a couple of people to Newcastle.

[James]

File format *and* protection...

What popular file format is commonly used by nontechnical people to hold tabular data and offers "password protection" on the files?

Yes, Excel.

[Anonymous Coward]

Open Access

Unless things have changed in the past few years then this is just the tip of the iceberg, and after having done some work for them in the past I'm not surprised

Try using personal information from the system to send letters asking for Autographs from the celebs.

Looking up family tax info, and amending them for better tax breaks.

Checking out your mates info.

Causing more hassle for your ex by amending their account for "EXTRA" checks or lower tax breaks.

Chasing up people's addresses at whim.

And these are just some of what HMRC get upto during work hours.... imagine if someone actually did their work instead.

IIRC, you can also search for partial matches on ANY field. And there are lots of fields..... want to know how many people have a HSBC account sure just stick in the first couple of sortcode digits and let it run.

Posted Anonymously from a cafe of course......

[Peter Johnston]

In the hands of criminals

HMRC states the data hasn't fallen into the hands of criminals. Yet they admit it has gone to KPMG. Any Private Eye or Register reader knows what that means. At least they haven't admitted that EDS has got hold of it - yet!

[Anonymous Coward]

The real costs

- about 47 coppers on the ground, doing a fine-combed search of all likely places these disks could be: tens of thousands;

- everybody in the HMRC foodchain working (paid) overtime to cover their ass: tens of thousands;

- spinmeisters employed to deal with pack of bloodhound press: tens of thousands;

- cost to banks and government to "monitor all accounts for suspicious usage": tens of thousands;

- cost for the now inevitable massive HMRC IT overhaul project coming our way, courtesy of [insert random consultancy house full of overpaid paper MCSE's]: millions

- seeing the retards in charge getting away with it all scott free: priceless.

[Anonymous Coward]

The basic two problems - and the untold story

It's cute to go into such detail, but you're missing the basics so here goes.

(1) The UK government does not have the equivalent of the US American Encryption Standard (AES), a standard encryption technique, open, published and peer reviewed so that it's easy to embed it in anything that needs security across the whole of government. In short - there WAS no crypto that the junior could have used.

(2) The junior defaulted to the 'safe transport' assumption that used to be true for internal government post. The only problem was that it *was* no longer safe as it was outsourced, but the risk management model was never updated to take that into account. So we have an assumption of safety where none exists, aka "false sense of security" (a thing long lost with this goverment, but I digress).

So, ladies and gents, the many smart ones amongst you will thus have already deduced from the above that, despite public statements to the contrary, AFAIK NO PROCEDURES WERE BROKEN. That's right - the chap simply did business-as-usual. Cynics may observe that the results were thus also business as usual, casually endangering the lives of quite a sizable chunk of the population.

I suggest you watch the government twist and turn to avoid admitting that one basic fact because it will be one hell of an indictment for the 'leadership'.

[Chris Gray]

My two cents

If they are anything like our DBAs they will have refused to do any sort of query and just copied one of the overnight backups onto disk.

A company I worked for was charged £2000/day for technical consultants (of which about £200/day went to the worker) so I wouldn't be surprised if the £5K quote was for one days work.

Just how big is a database of 25 million records? Our database is 250 MB and I wouldn't call it a big database. Can you get 25 million records on a couple of DVDs?

[Julian Bond]

What really happened

The cost of getting custom reports done by EDS is so high that a few years ago, they got a data export routine done that drops all the data into an MS Access file every month or so. Unfortunately the office junior on work experience who set up the MS Access reports has long since left and the departmental server was requisitioned so now the staff just make up a couple of CDs and pass them around so they can work on copies of the data on their own laptops. None of the staff actually know much of anything about MS Access or what's possible so when the NAO request came in they just burnt a couple more copies of the last set and mailed them off. Because some of the PCs in use are *old* this is also a pretty old version of MS Access. So when somebody claims it's password protected, what they really mean is that it uses one of Microsoft's laughable Office password schemes that can be broken in seconds.

Go on. Tell me that's not what really happened.

[Tom Chiverton]

That's all very nice

But the public don't read El Reg - if you've been smacking your gob over this go join you local No2ID branch, hassle the public on a few flyering sessions and make sure next time it's not 60 million records lost.

[Anonymous Coward]

Ah, the joys of outsourcing :-)

This 'subsetting' could have been done in no time at all had it not been outsourced to one of them there big consultancies. Insisting on sending someone data that they have expressly said they don't need is just plain wrong, wrong, wrong.

[Dave Harris]

It should be possible to release at least some information on the file encryption...

... as it should be secure enough that, for all practical purposes, it wouldn't matter. But I doubt that it was. Hence they aren't saying or, more to the point, daren't. Not to mention WTH are they doing sending it on CD's at all? Have they really not heard of telecommunications?

[Mike Stephens]

Stop Speculating!

What we do know is EDS provided the extracts and they were stored as 100 Zips. As far as I'm concerned that is encrypted despite what all these idiots are saying publicly. The protocol was the discs were sent without passwords. The recipient would ring up/email upon arrival and the sender would then email the password. This is consistent with my experience of HMRC. They were using this dataset because they couldn't get approval to spend money to get only what they needed. This is of course the enormous price you pay for outsourcing. What you could do in a couple of hours previously, now is almost impossible because of all of the layers of bureaucracy that snowball out of control once you get the likes of EDS, Capgemini, Accenture involved. It seems the single mistake here is the HMRC bods didn't imagine the enormous downside of the rather low probability risk they were taking.

[Neil Woolford]

And while you're at it, extract 100 tuples at random...

I know that there are reasons that the NAO (as people who should have some stats abilities) might want the whole 25 million lines so that they could check the sampling method for bias when taking their 100 samples; they don't just want the first hundred in the table, might wish to apply weightings to the selection for some reason and so on.

But having said that, it does seem overkill to ship a 25 million record database when the recipient is only going to use around 100 records. Even simply extracting every thousandth line would have probably given a useable sample to sample...

[Simon Ball]

Format

I'd guess on .mdb. Basic fomats like XML and CSV don't have passwords, and I can't see HMRC bothering to use an external programme to wrap them up. As for Excel - it's not possible to password the whole workbook without using encryption (though the default isn't very good). .mdb has a file password but doesn't use encryption.

[Anonymous Coward]

Excellent article

Valid points well laid out.

[Anonymous Coward]

Passwords and encryption.

I wonder if the term "password protected" is merely spin implying that the account details by themselves are not sufficient to access bank accounts and that a password is used. That is the password is used by the banks and nothing to do wth the data.

As regards encryption. RSA is not used for bulk encryption. A symmetric algorithm such as AES or triple DES is typically used with a random transport key. It is that key which may be encrypted using RSA or derived from a password.

[Sceptical Bastard]

Lies or simply ignorance?

Firstly, in several reports (including Newsnight's) HMRC stated clearly that the "the data was password protected but not encrypted". As Mark Whitehorn's piece implies, it is difficult to know precisely what the government spokesperson means by "password protected" because they evidently don't understand what they are dealing with - however, I think we can rely on "not encrypted" to mean, er, the data content of the files was not RSA encrypted.

Secondly, I too wonder what sort of database it is that cannot be queried by row or column or other defined field. The answer may be found in various reports mid-week and is alluded to above: the HMRC database is actually administrated by EDS. So it is EDS who decide what can or can't be done and at what cost.

If my experience of EDS's work for national quangos such as the former UKCC (the nursing regulator) is anything to go by, a sizeable floor area of the HMRC's publicly-owned building probably houses a motley crew of EDS doids who are contractually (and, no doubt, technically) the only people who can admin and manipulate the data - and, boy, does EDS charge for that work!

As an aside, I found similar Spanish practices when, many years ago, I investigated the disasterous EDS Child Support Computer System (CSCS).

That is why it is being claimed by government that it was too expensive to select the data as requested by the NAO. What they actually mean is that EDS would have charged through the nose.

As to the idea of electronically transfering the data over a WAN or the internet, that misses the point - it was NOT necessary and not desirable for the whole database to go anywhere! The subsetting and random selection should have been done in situ at HMRC, Washington. If necessary, that could have been done by an NAO staffmember (and for £5,000 he could have been flown up first class) then the resulting anonymised subset should have been RSA encrypted for secure transfer in the personal charge of the official to wherever it was to be audited.

My conclusion is that the public sector has yet to understand digital data; and that no government department (nor most of its outsourced 'partners') is competent to securely process digital data. This incident and its many predecessors demonstrate that just about every one of the government's and civil service's assumptions and policies and procedures relating to data handling and data security are flawed, misguided, based on ignorance or simply careless and ill-thought-through.

Furthermore, we (as taxpayers) are being royally ripped off by sharks - IT consultants, IT contactors and IT service-providers.

Whichever way you look at the current situation, it's a bloody shambles and a complete disgrace.

[Nano nano]

Is Database 101 ...

in any way related to Orwell's Room 101 ...?

[Calmfeet]

Requirements and Specification

For any significant system,software and the surrounding (human) environment, there must be a statement of requirements, prepared by the potential users before commencement,and a specification, the implementers' commitment as to what they will produce. Either of both of these may be updated as the work of implemetation progresses.

The statement of requirements should address questions such as the levels of security required,ease of extractability of information fields, and much much more,

without reference to specific details of implementation, since these are within the competence of the implementers, not that of the potential users.

What we need to see is is the statement of requirements and the statement as to the extent to which these have been met, but not how, in the implementation.

Such an examination will clearly reveal the level of human responsibility.

For example, if there is no adequate statement od requirements.....!!

[Slaine]

IQ test - 101

You can Trust your government

Your data is Safe

There has been no infraction of the data protection act

ID cards are a safe and efficient step forwards in combating crime

Outsourcing is clever

The Postal Service is reliable

One of these statements is true.

[jon]

emails reveal more....

think the emails released by ministers reveal that

1. they were very likely multiple password protected zips

2. the db extract was already done for another internal purpose (in whatever format) so sending a copy of that required no IT cost - hence was the path of least resistance.

Agree with Ray that getting even half a day's IT spend approved is going to involve a mass of bureaucracy.

emails: http://www.bbc.co.uk/blogs/nickrobinson/2007/11/those_emails_in.html

[Anonymous Coward]

The Frightening Thing About ID Cards

Watching David Davis on TV this morning brought it home to me. ID cards are a very bad thing.

These CD's being stolen (or just lost is more likely IMHO) is bad enough because you have something of value. But what about a situation where a terrorist organisation wanted to get access to somewhere? Stealing someone's fingerprints or DNA isn't really a choice.

However, what would happen if said terrorists REPLACED your data with their own, proving that that person was actually you? From that point forward you wouldn't be able to buy a loaf of bread in Tesco's. And government wouldn't want to correct the data because you clearly aren't who you are pretending to be.

That for me is the big danger with ID cards - giving the terrorist community the facility to masquerade as someone else very easily. That simply couldn't be done so easily if they had to produce a gas bill with a driving license.

Andrew

[Anonymous John]

Re In the hands of criminals

"Yet they admit it has gone to KPMG."

The NAO admitted that later. No reason at all for HMRC to know what the NAO did with the disks.

Whatever it takes to gets to get to the top of the Civil Service, it isn't an understanding of data security.

[simon]

I'll see your £500

and do it for a couple of pints over lunch whilst it runs ...

cut -d\, -f 1-2,4,6 db.csv > NAOout.csv

your delimiters may vary ...

[Anonymous Coward]

Rational Spending

Until recently I worked at a government agency (the RPA) as a consultant. I got yelled at a few times for doing things that should have been done by the contracted "IS partner" organisation, via the nebulous ever-changing, extraordinarily complex 'change process'.

But we knew, from painful experience, that something that would take me maybe two hours would take maybe four months and cost no less that £30,000 done the 'proper' way - and in actuality would not get done at all because there were other things it was more rational to spend £30,000 on.

Mostly I feel sorry for the 'Junior Official' at HMRC who was just trying to do what he'd been told to do - trying to do the job properly and with appropriate security would have meant never getting it done at all, and probably being regarded as simultaneously useless and trying to operate outside ones grade by 'management'.

Oh, and I feel sorry for the farmers. I never, ever, EVER want to work in government again.

[MGJ]

Encryption

The other main problem (other than EDS costs) is the GSI, where you can't use encryption (firewalls require to be able to read all attachments to virus check them) and there is no PKI available. Of course the NAO are not on the GSI so only low grade material can be sent to them electronically.

[Anonymous Coward]

Unresticted access?

Little attention is being given to the fact that some civil servant has _unresticted_ access to the entire database. With access controls like that you may as well assume that multiple copies of the database (of various accuracies) are floating around government departments all the time. Now that is worrying...

As for the use of the post, I guess that both sites have network firewalls installed and the necessary procedure to establish a secure network connection between the two was just too complex :-)

[Daniel Grout]

Re:File format *and* protection...

Excel is limited to something stupid like 65k rows (16 bit int size) so I doubt it was in Excel. That would mean approx 385 worksheets!

MDB - maybe - but PW protection is a pain in access and easily circumvented - what with MDA files and the like.

My money is on a huge csv (we regulary load in csvs up to 11 mill records so it is done) that was then zipped and spanned across disks; with the zip providing the much vaunted "password protection".

[Brett Brennan]

@ Sceptical Bastard, Anonymous (pirate) Coward and James Joy

You guys - collectively - have the reasons laid out: no one did anything "wrong", but things DID fall through the cracks.

The issue behind this whole mess is that NAO needed a representative, random subset of the data for their processing.

The usual method for doing this is to use (a) a random SELECT function that grabs X rows randomly, (b) a statistically tuned SELECT function that grabs X rows randomly that match the overall demographics of the database itself, (c) a complete database extract that is then analyzed by several methods and the appropriate subset extracted.

In my past work doing this sort of thing for a variety of customers (including the US Internal Revenue Service) the key to the extract is getting an unbiased extract based on the criteria of the auditing group. This *USUALLY* involves having a tech member of the audit group visit the source group on-site and supervise (or perform) the extract to insure that the unbiased extract is obtained. The data is then remanded into the possession of the auditor and removed personally by them to their location.

Sceptical Bastard and James Joy hit the main point here: for (a) or (b) above, you send someone up to the source site and have them get the data. If you can't do that, you do (c) - exactly what caused this problem in the first place.

Anonymous (pirate) Coward got the rest of the story: the SOP for (c) is to extract the whole database and send it to the NAO for their analysis. Inter-office mail is expected to be REASONABLY secure - after all, paycheques and HR info is routinely sent through this channel with no issue.

The one piece of data that's missing in all the press stories is the loss rate of the inter-office mail system itself. I'll be that the loss of this particular package is consistent with the overall losses in the system - this one just happened to contain a "political bomb".

Back when I worked in banking, we had a robotic inter-office mail system installed at our huge processing facility out in Brea, CA, USA. This consisted of little electric boxes about .5X.3X.2M that ran on a computerized "train track" through the entire building. The track was designed so that the little boxes could transit vertically between floors and up-side-down to negotiate other building obstructions.

Soon after the installation and operation of the system, a "scandal" developed when pay cheques failed to arrive in many departments after being consigned to the robot delivery system. An investigation was started, pay cheques were delayed pending "apprehension of the criminals", and all Hell broke loose.

Finally someone decided to check the bottoms of the vertical shafts and the plenum runs that the robots ran through upside-down. Lo and behold, the bundles of cheques were found, dusty but intact, at the bottom of one of the shafts, along with a lot of other "missing" stuff.

It turned out that the lid catch on several of the robot boxes had been damaged, allowing a heavy load to fall out. Once empty, the lid would latch again - no one the wiser. Paycheque bundles were "heavy" loads, as were computer tapes, six-packs of Coca-cola and several other items that were routinely sent between departments.

Several people were fired over this, mostly those sending soft drinks through the mails...

Yes, those disks are GONE for good...probably stuck in a crack in the back of the sorter...

[George Danezis]

Security should be discussed!

> Of course, "they" won't tell us and, in fairness, they shouldn't.

This is a dangerous and outdated view of computer security. It is well understood that how systems are secured MUST be the subject of public discussion and review. The security of live systems should rely on few well understood secrets (like keys or passwords), and not ignorance of the security architecture.

This is key to the development of the fields of cryptography, and security engineering that are taught and discussed in public, as well as the security of free source software that is open for all to inspect.

The government is clearly trying to say as little as possible on the matter, with good *political*, not security, reasons. It is unclear why IT journalists should play along with this strategy instead of asking for the full requirements, specifications, and even security audits of the systems that were involved in the data leaks. Making such documents public should not make the system more vulnerable, if it is engineered with security in mind.

George Danezis

(Security Researcher)

http://homes.esat.kuleuven.be/~gdanezis/

PS The idea that ignorance of the database format, or even the encrypted archive format, would slow down even an amateur attacker from retrieving the data is particularly silly.

[Giles Jones]

XML

Have you not heard of eGif standards?

http://www.govtalk.gov.uk/schemasstandards/egif.asp

It mandates XML format.

[RW]

Root Causes

This whole fiasco is the result (imho) of a pervasive problem in modern management "theory": the idea that it doesn't matter who does the work, that workers are all just interchangeable cogs, and are totally fungible. This theory is never stated explicitly, afaik, but holders of MBA degrees demonstrate its existence (and widespread application) daily.

The net effect of this theory is the devaluation of experience, expertise, intelligence, education, and inborn ability. Among other specific results, you end up with call centers with employees whose accents are too thick to be understood, convicted criminals having access to confidential financial data, workers who are simply unqualified to do the work at hand, and the surrender of control over important data to consulting firms.

Applied widely and indiscriminately, the theory of worker fungibility has a great many other consequences -- corollaries to the theory, if you will. Identifying these corollaries and relating them to the details of the HMRC data loss disaster is left as an exercise for the reader.

[Maurice Shakeshaft]

Ah, yes & no.

We don't know the disks have been stolen - we do know claims have been made about loss or non-receipt and so criminal activities are still speculation.

We don't know what service was to be provided for the cost quoted - we do know that it appears to be practically extortionate given what appears to have been asked for.

Some points:

This was a blunder waiting to happen - its just the scale and consequence that are outstanding.

Electronic Data is not currently (and cannot be) "safe" in the hands of either government or its IT contractors with the current systems and procedural models and practices. It is not, systematically, possible.

Every time there is a blunder the taxpayer pays. Even if it is made by an IT contractor. We are the only people who can pay. We are the "customer".

If a regular company blundered on this scale then it's likely that they would either go out of business due to loss of custom or would end up in court and paying punitive fines. Certainly, customer sanctions and legal processes could/would be applied. What chance here?

This blunder will not deflect the government one degree from ID cards because ID cards are a panacea for security ills. This blunder is not a "security problem" it is a "data processing problem". I've just written this and I don't understand it either!

The government need not care because there is no choice. It is, like its IT contractors, a monopoly provider. It may say it cares, it may say it will change - many partner-abusers do. Evidence is all. To others who know - do they change?

Let's see what happens when/if the status of the two disks is confirmed.

[Anonymous Coward]

re Root Causes

I have to agree with RW. In my experience most companies pretty much despise their techies , boffins and propellor heads.Hence the mad rush to outsource IT because the managers hate having people working for them who are more intelligent, more honest and more sensible than them.

Most organisations that are not run by the person who actually owns the organisation are run by idiots who have no respect for any kind of specialist ability beit artistic, scientific or technical. (because if they actually had any entrepreneurial ability they would own their own company). You can see this in so many organisations - for example the recent farce over the privatisation of Qinetiq where the people who made the big money are those who wouldn't know one end of a rail gun from the other.

Ah well, at least when this civilisation falls apart my descendants will know which end of an antelope thigh bone to use to hunt down those wildebeest. And how to make fire without a GANTT chart (unless its to provide kindling).

I'll get my coat (well anorak).

[Anonymous Coward]

It is simply no IT problem

and no problem of EDS, CG or any other outsourcer as well.

Sorry for me stepping in here rudely from germany, but after reading this, the problem was again the human factor. The HMRC guy who replied to the NAO: "You asked for all the data initially" brings it to the point: Why do you NAO guys bother me at all?

As far as i know, the NAO (or their alikes in other countries) do not have a fan club in the public sector, so pushing back on them is "natral" behaviour i guess.

This whole problem would definitely happened as well with an insourced IT department which would only have to be badly enogh aligned with the "business" .

What made me wonder though is: I have not heard about anything like this in germany, does that mean we never loose data / notebooks etc ????

[Chris G]

S.N.A.F.U

It is interesting to see that everyone of the technical commentators here have made one serious error, that there is some degree of professionalism in the civil? service and the out sources they use.

As regards most of the civil? servants I have met , they without exception feel put upon when asked top do anything that is not part of their normal routine. I can sympathise as having to put down your tea to process some work means when you pick it up again it has cooled down past it's optimum drinking temperature thus spoiling it and necessitating having to go and make another one. Further, when looking at outsources, they have to compete for the work and tender for their contracts at much lower rates than they would like, this in turn means that to maximise profits they are `forced´ to use cheaper labour. The old saying about paying peanuts comes into force. Professionalism is something rare nowadays, it does still exist but not in government offices, the people there generally look down on the population at large as THEM the moaners and the mob that doesn't appreciate civil? servants so don't expect things to get better other than the the ability to cover up. The only answer to cock ups like this is better training and management, something you will never get as long as the structure of the civil? service is as it is. Time for a decent revolution I would say.

[Matthew Johns]

Simple equation

Surely the cost of deleting the relevant data fields must be less that the cost of a train ticket to Newcastle and a day's time for a NAO junior auditor. Otherwise they would have sent someone down to do the sample selection on site. Therefore, pretty cheap. But the cost to the government? Maybe five years in power...

I've seen this set up before a few times. It is usually someone junior who's fouled up but this is invariably caused by the leader not caring about proper control systems (e.g. Chancellors who care more about results than how you got there...)

[Dom]

re re re

The only reason I can see for using TNT over the Royal Mail is to track the letters - but the silly people didn't! What's that about?

As Anonymous Coward (Sunday 25th November 2007 11:03 GMT) said, the default principle is that internal mail is safe - that's why it's worth paying for real employees to wheel carts around your buildings, and make night runs between your buildings with vans.

The default should be that it's not necessary to register internal post - seal it, because there's no point tempting people, but you don't need to register it because the mail room does that, and as they're delivering it anyway...

Can anyone tell me what's left to outsource in the British gov?

Not the royal family, that was sold to Hanover some time ago; not the PM - outsourced to Scotland...

[Anonymous Coward]

AES - American Encryption Standard?

Since when has the AES been the American Encryption Standard? Whilst the algorithm's competition was run by the US the A stands for Advanced, not American and A is not for Apple.

[Mark Whitehorn]

Re: Security should be discussed!

George Danezis makes an excellent point about good, well designed security systems. As he says “Making such documents public should not make the system more vulnerable, if it is engineered with security in mind.” I agree.

In general, the less you know about a security system, the more difficult it is to break. However if, as George suggests, a system is well engineered with security in mind, it possible that some information about the architecture can be revealed without compromising the system.

But the converse is also true. Some badly designed systems rely on the fact that the architecture is hidden to provide some of the security. I’m not suggesting that this should be the case, merely that sometimes it is so.

For example, imagine a physical security system that includes a wire on top of a wall. If you know nothing about the wire or the signal it may carry, you risk detection if you cut it. On the other hand, if you know that it carries a very simple signal that can only detect a complete break you can happily use a jumper wire to avoid detection. (I don’t write from experience here, but I have watched innumerable spy movies).

Now it is clear (painfully, excruciatingly clear) that the system under discussion was not well-designed. Had it been, we would not be discussing it. And given that it was poorly designed, it may be that some measure of protection might still be afforded if the remaining details of the ‘architecture’ are not revealed.

I agree that ignorance of the database format or even the encrypted archive format will not appreciably slow down professionals. How much it would slow down or stop amateurs would depend upon their level of expertise – which is unknown.

Clearly there is a spectrum of risk here. Some information (the file names) is very low risk, other information (the password) carries a somewhat greater risk. Exactly where the line should be drawn is tricky but the government is wise to err on the side of caution. Doing otherwise has the potential to further compromising security to an unknown degree.

>The government is clearly trying to say as little as possible on the matter,

> with good *political*, not security, reasons.

I agree. Given that the government has, in the past, shown very little concern about protecting this data (hence the leak) there is every reason to believe that it is currently more concerned with the politics than security. But the motivation of the government and the morally correct course of action are not linked as cause and effect. In other words, just because the government has a hidden agenda for wishing not to discuss the details of the security does not mean that those details should be discussed.

>It is unclear why IT journalists should play along with this strategy

>instead of asking for the full requirements, specifications…

I don’t agree that we are playing along with a strategy. I think that journalists face the same choice as the government (but without the political pressure) and, for the reasons outlined above, should make the same decision.

I suspect that if we were ‘playing along’ with some government strategy, we wouldn’t be highlighting the absurdity of that same government using pseudo-technical arguments for political ends.

[Lord-a-miytee]

ID cards, Unique Identifiers.

So ID cards make data easier to tie together because they use a unique identifier, eh? What a spiffing idea. Especially as i've already got one of those. It's called a "National Insurance Number". In fact, i have another unique identifier. It's called a "National Health Number". Now i'm going to get a "National Dupe Number" too?

i'm leaving this farce of a country as soon as i have the wherewithal.

[Nigee]

process?

There seem to be some other points, namely UK Govt security procedures for information that is not National Security related. Do these exist? Is there a classification regime? Is there policy that relates privacy requirments to security classification and associated procedures? Are there handling procedures covering the requirements for the different media and each classification level? A simple and reasonably pragmatic example is at http://www.gcio.nsw.gov.au/documents/Labelling_Sensitive_Info.pdf

[Anonymous Coward]

ID cards and the Irish

Thinking about ID cards: I was talking to a friend who is entitled to an Irish passport.

He/she said that if ID cards were introduced, the whole family would immediately activate their Irish citizenship.. and refuse to carry ID cards.

What proportion of the population in the UK would be able to do the same?

[John Stag]

Blame XML

What if some government-paid twit had decided to store the data in XML format, as is the fashion?

That would mean it was hard to strip out the bank details, etc.

[Brian Squibb]

Re: ID cards and the Irish

I guess the 3 million immigrants this government has allowed in the last few years could also wave their 'other' passports and not carry ID cards.

[Daniel Grout]

Blame XML

It wouldn't be hard at all to remove data from an XML file - it would probably be even easier!

[Jeremy Hooks]

@According to yesterday's Telegraph

>> The NAO (not having a mainframe) then passed the database to KPMG to process. -- Anonymous John

That is an interesting addition to the story. The only reasons that I can see the NAO would need mainframe to handle 2 CD worth of data are (if they only wanted to process 100 random records):

1. Their desktop PCs are steam powered.

2. The data is in some sort of raw format which can only read by an obscure DBMS, which only runs on a mainframe.

[Anthony]

Scary

In todays ultra-technologically advanced world where fighter pilots can shoot down their targets using in-visor overlays and a nod of the head, where infantry can track down the scrambled and secure cell phones of foreign dictators, where apparently Israel can sneak past Syria and maybe do *stuff*, where our personal data is at its most vulnerable as the Government seeks to consolidate that data and store it in one place..

In this society, the latest scandal isn't that our data was hacked and sucked off into a torrent file, there was no IT security failure in terms of hardware breaches or software cracked, there was no inside man handing out passkeys or ID badges or flicking power switches and blacking out surveillance. None of this. The Government simply put the data onto a CD and put the CD into an envelope.

I suppose we should be grateful that the NAO didn't just ask Alistair Darling to empty all our bank accounts and send them the money as from the sounds of it he would have done.

"Hey, exceedingly junior scapegoat, stuff that money in that envelope and send it in the post"

"But sir.."

"Now, now young chap, time is money, on with the stuffing!"

[Anonymous Coward]

Re: ID cards and the Irish

I don't believe that will help them at all (unless they move back to Ireland) as the Government plans to force *everyone* living and working in the UK to have an ID card. About the only people that will be immune to this gross invasion of privacy will be foreign diplomats and the Queen.

The whole ID card thing is truly frightening because of two opposing issues:-

1. Just like any other Government IT project, the whole thing will be an absolute disaster and the highly sensitive and personal data the database contains will be about as secure as a chocolate fireguard.

2. The Government believes ( or is, at least, spinning that )the ID card system will be secure and infallible.

Put those together and the opportunities for miscarriages of justice are immense.

[MrT]

Pass the parcel...

"The NAO admitted that later. No reason at all for HMRC to know what the NAO did with the disks."

NAO commentator also went on to say that the data was HMRC's, and that HMRC had a duty to care for the data. NAO therefore consider passing on the data to a third-party for processing to be 'business as usual'. It's possibly true that HMRC had no knowledge that NAO was going to pass it all on, but NAO have also been irresponsible here since it was told by HMRC management that the data would not be desensitised.

Oh, and on that matter, 'desensitised' is not the same as removing fields, and may mean recoding certain parts of citizens' records in a way that allows whatever statistical analysis NAO had planned to actually be carried out. It takes a short time to remove entire fields from an export file, but a long time to replace information that preserves uniqueness of, say, the NI number or the postcode/house number combinations without giving away information that could be used to directly link an individual - as I understand it, NAO was auditing the HMRC, not trying to find benefit cheats, so they needed to know that HMRC was on top of the potential for fraud, rather than being able to hand back info about what X, Y or Z were up to... (apologies if they are monikers anyone here uses - I was going to put 'Joe Blogs'...!)

I think here there's another issue - govt depts not being clear with each other.

And there's one more cost to the country that was missed from AC's list - I've already recieved a letter from the acting head of HMRC to tell me to not panic. - 7.5 million letters at 20p or so (probably less due to the bulk mail - IIRC it could be 16p each after discount) has already been spent on spin.

[Anonymous Coward]

Do not blame XML!

> What if some government-paid twit had decided to store the data in XML format, as is the fashion?

Not likely. The data should be nicely record-oriented, so using XML is Bad Practice, not to mention unwieldy. It is most likely in some RDBMS.

> That would mean it was hard to strip out the bank details, etc.

That's not a "hard" that I recognize. You can use XML extraction languages like XQuery to get at the data, or you can roll your own with some Perl, a task in the order of an afternoon's work (until everyone is satisfied).

[Name]

Re: ID cards and the Irish

Except, of course, immigrants will be the first to be given ID cards starting from next year. Still, never let the facts get in the way of a good Daily Mail rant.

[Scott Broukell]

KPMG deleted their copy

Well, at least KPMG's deleted files are safe as houses in the "Recycle Bin" on their system. I'm sure they emptied it after the deletion to make sure it was unrecoverable.

I sure hope they ran an multi-overwrite and put plenty of garbage over the freed-up disk space after the deletion?

Surely such an outfit would know what to do to make data recovery very, very difficult when the next refresh sends their current kit off for auction!

[Anonymous Coward]

Re: IQ test - 101

"One of these statements is true." - Oxymoron

[Risky]

Strategic

What's the betting there's a huge team working on the 'stategic' solution of rthe database, with the new system due in "6-12 months". Meantime all you have a creaking access db to do the days work, the DBAs telling you you can't have anything, ever and the strategic system producing nothing but powerpoint.

[Anonymous Coward]

The ID Card database would be much worse

Losing the ID Card database would be even worse than you describe. No need to reverse engineer the fingerprint codes - just look for any that are close-enough to your own. Large scale trials of the US-VISIT IDENT system found about a 0.1% false match rate, so in a database of 40 million adults a typical crook should find around 40,000 people to rip off at leisure (complete with names, addresses, passport numbers and all).

[Ashley Stevens]

Inter-departnmental politics

I'm surprised no one has spotted the obvious inter-departmental politics in the response that removing unnecessary data was too costly. The NAO website says "The National Audit Office scrutinises public spending on behalf of Parliament.....Our work saves the taxpayer millions of pounds every year."

The response that removing unnecessary data was too costly was obviously a political response. "If you're so worried about costs to the public, you deal with it." It was obviously meant as a polite 'up yours' to the idea of creating more work and hence costs (however small) in order to perform the audit. In other words, a politically expedient way of avoiding doing extra work.

[John K]

The combined resources....

HMRC may well outsource their IT, but are they saying that from the combined resources of both the Inland Revenue and Customs & Excise, they don't employ one person capable of stripping out a bit of data from a system they own?

[Simon Arthur]

The undicssued issue

Yes, I am _really_ annoyed that the NAO has lost yet another set of sensitive data. A similar thing happen with employee data earlier in the year.

But the big question, which no one seems to have mentioned, is why are the allowed to request the data in the first place. Isn't the Data Protection Act to protect and prevent the disclosure of personal data to unauthorised bodies.

In local government, we're not allowed to share data with other departments or authorities without notifying the named persons in the data. You need there consent. Why is it different for central government??

[Ian]

No DBMS?

Are you people telling me the data wasn't even stored in a relational database system? Have these companies never heard of MySQL, PostgreSQL or Oracle?

[threaded]

Encryption?!

As someone who knows a little more than nothing about encryption I'd like to point out that even if they'd encrypted these disks you should still have been worried.

This data is going to have significant value for many years. In fact it will only start to make good money for the bad guys in about 3 years, and then onwards for a lifetime, what's that, maybe another 80 years more. If bad guys have got them they'll probably sit on them for a good few years before even starting to use them.

Cryptography is always advancing, and so is the speed of machines. Encryption systems in use today will be broken eventually, they always are. These disks have a significant value, and will continue to do so for a long time. It would be worth the time and money for the bad guys to break the scheme in use. (I now look forward to myriads of posts about how hard it is to crack the current encryption schemes. Yes, currently it is hard, but next year it will be easier, and in ten years: probably trivial.) Considering the governments wherewithal on security, I doubt they would have encrypted it properly anyway, even if they'd tried.

The issue is not the use of CDs, the posting in the mail, or the lack of encryption, the issues are these lunatics thought it OK to send a large quantity of that quality of data about, as it exhibits a monstrous level of cluelessness, and that people so cavalier are even allowed through the gates, never mind given positions of authority.

[David S]

@ Scott Broukell

Well, having worked for KPMG, I can say without doubt that there are people there who WOULD know all about data security, and how to securely delete a copy of a file. Their Forensic Accounting department, for example, frequently had to recover "deleted" and even "overwritten" data. They were quite the impressive bunch. Probably downsized in the interim, of course.

On the other hand, my abiding memory of KPMG was being called to a senior partner's office because he was having trouble opening a Word document. A short explanation of the difference between double-clicking (cli-click) and clicking twice (click click) was required.*

I rather fear that this would be the level of numptie with which these data would have been entrusted...

* Of course I can feel as superior as I like, but at the end of the day he was "earning" six figures for not knowing how to double-click while I, with all my wit and sophistication, struggle to support a family and a mortgage. Who has the last laugh?

[amanfromMars]

Fluff and NonSense? Use Imagination ....

If a DataBase had Total Information Awareness of a Citizen's Needs for the Future he has Seen to Share, IT would Allow for Government Payment of Public Money to a Citizen who has Shared Everything for Transparency to Liquidate Valid Future Costs/Past Expenses.

And all apparently for a measly seven seven figure sum. QuITe obviously the powers that be are not au-fait with the Power in Miners of Rock.

And boy, are they in for a Pleasant Surprise Package? Not 'arf.

"Reality leaves a lot to the imagination." .... John Lennon.

[Anonymous Coward]

U.S. Baby Boomers

At least in the US, when the Baby Boomer generation is all dead, we will finally have politicians who have SOME grasp of technology.

[TeeCee]

The *real* insight here is......

"....... the fact that our government has demonstrated a complete lack of ability to protect our data is, for me, a strong argument against ID cards. But then, I'm not a politician."

And there we have it folks, the ID card problem in a nutshell.

[Anonymous Coward]

Expect more of this in the future

The problem was summarised very well in the first comment. EDS is to blame. Why is nobody pointing the finger at them?

The bigger picture is that these problems will continue to occur because the incompetent/corrupt/stupid/lazy buffoons that award the contracts for these systems only seem to rely on the response to one question: "Has your organisation done anything like this before?".

Only the usual suspects can answer "yes" to this questions, so only they get chosen. It doesn't seem to have occurred to anyone to ask "How badly did you f**k it up last time you did something like this before?"

[Marty]

Why dont you bame it on....

I cant belive there are so many comments on this artical and nobody has blamed it on Bill Gates.....

I mean.... it must be his fault....

[Scott Broukell]

The Passenger (IggyPop)

Thanks David S for the insight. I kinda guessed that would be the situation. BTW – knowledge is of far greater value to the world than money, but harder to pay bills with :-(

However, how about the ability to designate certain fields as “confidential” in such a way as to lock them down, make them non-printing or non-exportable or whatever. I mean like ”Admin” rights allow, or deny shares, edits and read/write actions on files. I imagine that the makers of decent software have thought of this one? Kind of a built in automatic filter that simply won't let the entire data set be copied/cloned without the intervention of an authorized “owner”, of suitable seniority and nous, who would have to carry the can if things went wrong because their details would be embedded in the data set. Of course it all comes down to human actions and ability levels and there's usually always a “work-around” somewhere I guess.

We are all passengers now on the information super-highway but we don't expect the trolley-dolly to be flying the plane when we are at 32,000 feet over the Atlantic! We would all refuse to fly, I suspect, if we thought for a moment that might be the case. But how can we refuse to be swept along at 100Mbs, hurtling towards inevitable disasters, such as that of HMRC, because there's no-one at the controls actually! Scary, stop I want to get off.... now!

[Anonymous Coward]

Did you post or e-mail your cv to them, Mark?

A very lucid explanation. Thanks Mark.

[Anonymous Coward]

What lies behind ID cards - and why it's relevant

You're all talking about ID cards as a Bad Thing (tm), which in its current incarnation and planned use it surely is.

However, the issue and control of those lovely National Insurance/Health/whatever numbers has been so totally botched that anyone who has looked at it over time has declared it non-fixable.

There is a HUGE amount of benefit fraud performed by the use of the cracks in the issue system, and thus one of the non-Orwellian drivers behind the IDcard was to redo the body numbering from scratch. It doesn't excuse the rest of it though..

[Anonymous Coward]

1 in a million

- I wonder how many times data has been sent on CD's and arrived safely.

- I wonder how many times data has been sent on CD's and forgotten, never to arrive and just getting lost in the internal mail.

It is good that this information has gone public. It could of been very effectivly covered up. For that we should give our Government some credit.

Also, out of the 1000's of data files that get exchanged within government, it was going to happen one day. This is the price we pay for storing sooooo much data in one place.

[amanfromMars]

Follow the Leader of Systems or the Driver?

"I imagine that the makers of decent software have thought of this one? Kind of a built in automatic filter that simply won't let the entire data set be copied/cloned without the intervention of an authorized “owner”, of suitable seniority and nous, who would have to carry the can if things went wrong because their details would be embedded in the data set. Of course it all comes down to human actions and ability levels and there's usually always a “work-around” somewhere I guess."

Scott,

The owner of any decent software would intentionally embed all relative details in the data set so that it runs to specification. In fact, it is quite naturally included in every thought that we share/line of code that is written and decent software has broad enough shoulders and a thick enough skin to carry the torch rather than harbour any thoughts on carrying a can.

It is a subtle failing in programming, which may be intentionally placed there, to have doubt hinder ability thus maintaining a Moribund Status Quo Logic. A gift from you know whom.

SeXXXX IT, Billy Boy, Breathe some Life into the GAIme. In the SurReality of Virtual Space though, is Power Directly Proportional to Proxy Ethereal Control of MindSets with an All Pervasive and Addictively Persuasive Seduction ..... in Order to Guarantee Positively Reinforcing Results.

[Ben]

Re: What it costs vs what it costs

And other posts that suggest that this kind of extract should be cheap "I'll do it for £500" etc.

If it is done on the cheap, without the involvement of suitable governance such as the outsider will provide, then the results are all too apparent.

[Anonymous Coward]

Met Police and CapGem.....

Posting Anon for obvious reasons.....

Worked for the Met and still have contacts inside at High Level in Empress State/Cobalt Square/NSY.......

I was looking at Virtualisation for a project at my current employer and mentioned to mate about how good it would be for their department to Virtualise the servers...

Apparently they are not allowed to even think about changing/moving the databases and/or any of the servers they reside on as the contracts are signed for over 10 years..... Any changes they make have to be approved by CapGemini and also carried out by their engineers. The costs involved are staggering......

They are having a refresh of the systems there and the desktop systems they use for email ("AWARE") are being upgraded. Long story short they also have desktops sourced internally and an engineer from CapGem, rudeboy type with matching ringtone, was meant to be installing replacement machines where nessecary and dismantled a machine because he didn't understand why the machine wasn't working on AWARE and left the machine POST erroring due to removal of the memory and went home. Suffice to say mate found machine and it happened to be a machine that engineer wasn't authorised to touch. Can they do anything about it.... Can they eck.... Oh and if you happen to pay UK tax you paid for that idiot to spend the day breaking stuff.

P.S In case any are interested....The issue with the machine is that it was plugged in underneath the desk into a KVM so the user could switch and only use one monitor. The one he was meant to refresh was under said monitor. Oh and the engineer/moron gets paid over 30k pa for this.

[Anonymous Coward]

The moral of this fiasco seems to be ...

... DON'T OUTSOURCE

[Chris]

ORly?

May seem strange; but I don't believe for one second these 'disks' have been 'lost'; I don't even beleive they were sent in the first place. This ordeal is far far 'too' stupid for the Government.

I beleive this is a way to promote ID Cards, so people buy into it in a bid to 'protect our personal details'!!

They know exactly what they are doing, and we all just walk right into it; everytime!

[/conspiracytheory]

[amanfromMars]

Sub prime markets......crawling out of the woodwork

"Apparently they are not allowed to even think about changing/moving the databases and/or any of the servers they reside on as the contracts are signed for over 10 years..... Any changes they make have to be approved by CapGemini and also carried out by their engineers. The costs involved are staggering......"

An Inequitable and Unfair Slave Contract, AC,......and probably Illegal/Criminal for it would appear to guarantee Non Competition Complacency/Gravy Train Riding rather than keeping evryone at the top of their game.

Paying for Failure ... the New Labour Way.

[Peter Robinson]

Bad news for mothers

Apparently 7.5 million letters have been sent out (giving Royal Mail £1m they badly need?) and 25 million records were lost. So maybe 18 million children and 7.5 million parents (mostly mothers) are involved, and the mothers are the ones whose bank accounts are at risk.

Has any journalist or politician spotted this relationship?

Fortunately I have no young children, I am retired from IT and I live in France, but I sympathise.

[Philippa Sutton]

Baby Boomer technology

re Baby Boomers

Mr A Coward says:

"At least in the US, when the Baby Boomer generation is all dead, we will finally have politicians who have SOME grasp of technology."

And we will also have in power a generation where there is widespread belief that Earth is 6,000 years old, that the US never sent spaceships to the moon and that science is a godless conspiracy to lure ordinary folks into Satan's lure.