Technical issue?
PEBKAC
Estimates detailing the loss of criminal evidence records by the UK Home Office and the police show the figure leaping from 150,000 to 400,000. Nonetheless, in a statement issued on Saturday, the government department said it was working to restore the records using the magic of computer code. "Working with the National …
Competent system design would admit that PEBKAC is likely and should prevent permanent data loss - but this is only part of the problem. Data storage in this environment is complex, complying with legislation created by innocent politicians makes it even worse - particularly when the work is sub-contracted out to a bunch of programmers who's only experience with police data collection and maintenance is just a DUI ticket.
> "The incident highlights the fact that IT practitioners should be accountable to independent professional standards"
I've no doubt there's going to be an epic amount of recriminations in the wake of this. But I'd be wary of pushing all the blame onto the "IT Practitioners".
There's clearly two issues here.
The first lies in the processes which allowed a breaking change of this magnitude to get onto the live plaform.
The second lies in the backup processes (or lack thereof) which mean that they've been unable to recover the data.
In both cases, a significant factor will have been the amount of time/budget allocated, and the oversight from upper management.
And neither of those are "IT" specific issues.
E.g. I've worked in places where the response to rollout failures was to double down on process. The result was that things took much longer to roll out, and that the "coal face" IT people actually had less time to deal with the technical side of their job, since they were constantly having to complete paperwork and defend the need for said technical activities.
As a result, customer-reported issues remained open for significantly longer. If they were fixed at all - the time/energy cost of fixing some things simply fell below the level where the effort could be justified.
And at best, the number of issues remained pretty much the same. Since the new high-overheads process didn't address the underlying issues; they just slowed down the rate at which changes were rolled out.
There's always going to be a balance of some sort - there has to be. But the blame definitely doesn't all rest at the feet of the IT people.
"The incident highlights the fact that IT practitioners should be accountable to independent professional standards"
Notice that it was the BCS that said this. What a surprise(*), as they've always been trying to make IT a chartered profession so that they could charge for qualification courses and exams.
The only problem is that their certification will always be at least 5 years behind the current trends in software development, because of the time it takes to create the certification.
(*) Yes, I've always been bitter, because they did not accept that my degree did not even exempt me from the BCS part 1 entrance exams, so I never joined.
We used to get membership of the BCS provided by our work. As far as I can tell, this consisted of a regular (quarterly?) copy of a very dry and dull magazine, and the ability to go to some meetings and talks that nobody ever bothered with, so our employer stopped paying for it, in favour of more useful things like Pluralsight subscriptions.
At some point there was talk of scientists having to become members of the relevant charted institute so I joined mine whilst my degree counted for admission in case they made the MIBiol exam route compulsory (I'd long since promised myself no more exams ever). The talk came to nothing but what made the membership worthwhile was a section of the magazine. It turned out most/a lot of the members were teachers and there was always a selection of exam howlers.
"It is likely that a developer..."
Should ignorant organisations make statements of justification based on opinions? This doesn't sound very professional to me. I am a member of the BCS and am uncomfortable with this sort of speculation being used for justification.
@"or am I reading too much into this?" nah it is just BCS talking out of their Rs in the hope that they can shoehorn themselves into being seen as an authority.
I looked at BCS back in the '90s and discovered that someone with a MCSE qualification and 5 years experience was in their top 25% of memberhip abilities. This was the time when brain dumps had all the questions and answers availible for every MS test and there was no confirmation that the person holding the certificates was the same as the one who took the tests. Surprising how many people apparantly took their tests in places like india and yet proved to be clueless about the subject.
I personally didn't bother signing up with BCS to avoid being tarred with the same brush given my experience that actual expertise was not a requirement for membership.
I am not sure if backups are really a problem here. This process might have actually require there to be no backups.
The police WILL have records that they are not allowed to keep indefinitely and that will need to be permanently removed when certain criteria are met. Perhaps because by law they are only allowed to keep some record for X years. Or because the longer retention of some data is based on the outcome of court proceedings where a not guilty verdict might have a legal impact on retention of some records.
The police SHOULD have a system that culls any data that meets these criteria and that data SHOULD not be retained in backups.
The police MIGHT have a weekly automated job to delete all the records that have met the deletion criteria in the previous seven days. This process MIGHT have some manual requirements (because of a flawed design or because of a deliberate safeguard). If there is a human error involved this is where that might have happened. It may not be a coincidence that it happened at the start of the year. Perhaps someone accidentally changed something to the wrong year and an additional year of data was deleted. In a process that is meant to be legally irretrievable so no backups.
Just speculating but it would explain a thing or two...
I'd speculate that the thing that someone might have accidentally changed might relate to new requirements arising from political decisions to no longer being part of EU-wide policing, and the requirements to remove data relating to EU citizens outside of the UK that arise from that.
This would also explain why the likes of Priti Patel have been so vague about what the actual cause was, because the long-and-short answer is her and her brexity cohorts.
"why the likes of Priti Patel have been so vague about what the actual cause was"
The last time the HO tried to brief their front person on techy matters all she thought she could remember was something about hashtags. I guess they're avoiding going there again.
That is not an unreasonable assumption with regards to the timing of the incident. It would, however, suggest that the UK government is serious about not breaching agreements. And they have a poor reputation with regards to sticking to their word when it comes to sticking to data agreements
Why not? The incompetents in charge of Brexit have f***ed up so many other things, why not this one. Perhaps they were trying to delete data acquired from systems that they no longer have legal access to post-Brexit. Perhaps there is to be an audit process to verify that they have actually followed EU data deletion procedures, and someone panicked because they never have and that could invalidate the "Deal".
True, you can't blame everything on brexit, but we have left data sharing agreements with EU-wide policing, which would affect systems like this, and the timing is spot on.
I've no evidence that this is the cause, but it does fall into the "puppy sitting next to a pile of poo" category.
Blame brexit? Who would be daft enough to do that when it's provided all these wonderful advantages to us?
I mean, we wouldn't have ploughed ahead with a stupid policy with no obvious advantages, would we?
I think we should give the £250 million we're saving a week to our nhs heroes!
Not the foreign ones though, obv.
Jesus there are some stupid twats when it comes to the EU and Brexit. There are 4 things you should think of:
1. Good things about the EU.
2. Bad things about the EU.
3. Good things about Brexit.
4. Bad things about Brexit.
If all you know of and repetitively bleat on about is 4, then I suggest you get your head out of your backside and do some research. If you think the EU is all honey and light then get your head out of your backside and do some research. Put some nuance into your arguments and then come back from the proctologist because brexit brexit blah blah shows a real lack of thinking.
Indeed, 1 and 2 manifestly exist (although I've never seen any in the 2 category that were particularly convincing*, or which didn't apply more generally to our own system of government**, and which didn't seem like a reasonable trade-off for the benefits gained), and number 4 is becoming more apparent by the day. Some concrete examples of 3 please, or can we safely strike that one from the list?
In the 1970s, when we originally joined what was then the EEC, the UK was known as the "Sick Man of Europe", and joining the bloc literally saved our economy. That name is going to be applicable again really soon, and in more than one sense, if you look at the most recent weekly COVID death figures published today which give us "world-beating" mortality figures.
*I've seen plenty of whining about the ECB, the Euro, the Schengen borderless zone (but oddly, never the other aspects of the Schengen, like the police data sharing), all things which clearly did not apply to the UK when it was a member with its OPT OUT.
**Such as the "bureaucrat" argument, which conveniently ignores the fact that the EU has around 30K civil servants, as opposed to the 300K in Whitehall, and one of the points of 28 27 countries working together is to reduce bureaucracy through economies of scale.
Anyway, some examples of 3 please. Anyone? You, behind the tumbleweed?
The point is for you to do the thinking. Its a common character trait I've found that rabid supporters of the EU really like someone else to do their thinking for them. If its someone with a foreign accent to make them feel all special and sophisticated then even better.
If you think criticism of the Euro is whining then you need to shut-up or do some reading. It's fundamentally flawed and its supporting fiscal policy doesn't scale to the eurozone. Even the EU themselves acknowledge that.
If you think criticism of the Euro was a valid reason for the UK to leave the EU, then I suggest you look in your wallet and see whether it has Euros in it.
Before our free movement was taken away, I had the privilege of travelling freely to several countries inside the EU, and also within the common market but which are not EU members. Some of these countries have the Euro (France, Italy, Germany, Greece, Spain, Portugal), some do not (Czechia, UK, Iceland). The utility of the Euro is a matter entirely for those countries that use it.
It's a "common character trait" I've found for supporters of brexit to characterise those who question their reasoning as "rabid". It's almost as if they think an ad-hom attack on others will somehow strengthen their argument. They confuse "winning" an argument with bullying someone into submission.
Brexit is just the start of the collapse of the whole EU project. When it was a trading organisation (the EEC that the UK joined in the '70s), it actually helped the economies of its members. During the '90s, it morphed into a grand socialistic project, invented its own "currency" (which is backed by nothing!), and started to accumulate ever more members by signing up the ex-eastern bloc countries, all of whom were broke.....
The Germans and the French have just realised that they're the only ones putting money into the EU project, and most of it is going to prop up the failed economies of the Mediterranean countries. Those economies probably wouldn't have failed if they hadn't been forced in the Euro, but that's another issue...
The bookmakers are taking bets on the next country to leave. The "smart money" seems to be split roughly evenly between Ireland and the Netherlands. Ireland have realised that they're no longer getting the "development grants" (aka "bribes") and are now expected to pay into the limitless European Pit. The Netherlands just want out - they're fed up with being dictated to by Brussels (they fought wars to avoid that a few hundred years ago!).
My banker friends in the City of London tell me that the EU will be lucky to survive as it is into 2023. Many of the smaller countries will leave, and there will be the usual emnity between France and Germany, with each blaming the other for it all going wrong!
Currently EU fiscal policy and the EURO is governed by the German race memory of the 1920s hyper-inflation, they are applying remedies to fix a problems that occurred 100 years ago.
EU social policy has been high-jacked by the right wing zealots of Poland and Hungary -- and under the current rules there is nothing the other 15 members can do about it.
Interestingly both Spain and Italy have rejected the EUs Covid relief package after they saw what happened to Greece when they accepted a rescue loan from the EU. EU economists are still advocating "a course of leeches" as the solution to all problems.
Many good things abut the EU (its where I live) but it is very far from perfect.
The fiscal policy of the ECB had no bearing on us whilst we were members; we weren't members of the ECB or Euro. Tey may be valid criticisms of those institutions, and I'll not claim they are perfect. However, these were used as arguments for the UK leaving the EU, and they just weren't valid because they were totally inapplicable.
It's a bit rich to accuse the EU of having right-wing social policy when our own country has the most right-wing government in living memory. A government who are currently talking about removing regulations that protect workers from exploitation, and most recently decided that it is fine to do trade deals with countries that commit genocide. The EU members, on the other hand, are making quite loud disapproving noises towards Hungary and Poland exactly because their social policies do not align.
I'm not going to claim the EU is perfect; no system is. However, it is founded on cooperation (and compromise) which is better than the alternative which is competition and conflict. I don't think it was ever well explained to the people of this country why compromise is a price worth paying for peace and prosperity.
Germany wasn't the only country to experience hyper-inflation in the 20th century either, so calling this "German race memory" isn't exactly accurate. Austria, Greece, Hungary and Poland all had hyperinflation around the same time. It's largely Greece's fiscal irresponsibility before they joined the Euro, and dishonesty around how much debt the country had that led to the Greek financial crisis of a few years ago. Whilst I don't particularly agree with the response from the ECB of enforced austerity, it's not like the UK didn't have the exact same neoliberal economic policies over the last decade. This isn't a problem that is unique to the ECB, but it is one that comes around when political ideology takes over from economics, which itself isn't exactly a discipline that is free from murkiness and false assumptions.
"The UK requests and uses more data about foreign offenders than almost any other country. We are one of the biggest users of the European criminal records system and are leading the way in Europe to encourage the proactive sharing of information about individuals who could pose a risk to the public." - Home Office spokesperson, 2014
"Hi guys, one last thing before we go...uh this is embarrassing... do you still have a copy of our police datas? WTT three day old dogfish" - HO 2021
"The police WILL have records that they are not allowed to keep indefinitely and that will need to be permanently removed when certain criteria are met."
Nothing is ever permanently removed. Beat plods won't be allowed any visibility, sure. But senior police will probably be allowed to see that there was a record of something, and can get an ministerial exception to see the data. Cheltenham will keep everything, for ever. That's their job.
There's a long history of there being records they're not allowed to keep indefinitely, that have been kept indefinitely. The excuse being the difficulty of removing them because although they should have a process for removing them, they don't. It's been reported here a number of times. The admission that they have a weeding process blows that one out of the water.
I suspect the rationale being given for not previously being able to remove data would be along the lines of not being able to identify the records to remove in historical data. The reasonable response would be, "well, start recording the information you need to be able to do this in future," so that any new data can be subject to such cleansing requirements.
It's a bit like the argument for technical debt in legacy software - you might not have the resources to go back and completely re-write stuff that was written in the '80s in COBOL, that is working, but is unmaintainable. However, you sure as hell can follow best practices with anything new you write in a modern OO language that supports SOLID.
> I am not sure if backups are really a problem here. This process might have actually require there to be no backups.
A system with zero backups? Seems highly unlikely.
I mean, I take the point that there is (probably) a hard limit on how long data can be retained for. But equally, that should be the point at which it's removed from backups, not production.
E.g. if you have a 2 year limit on data retention, then the data should be removed from production after 18 months, and then fully nuked when the backups expire.
Whatever happens, there should always be a way to restore your production system with recent data in the event of a catastrophic failure. And by the sound of it, that's what's missing here.
Admittedly, this is the government, so I'm guessing the rules and policies involved are even more convoluted and unexplainable than Kafka on acid...
"
A system with zero backups? Seems highly unlikely.
"
Yesterday's article mentioned that there used to be a backup in the form of a complete mirror-server at a different location, but that was lost in a fire several years ago and so far not replaced.
In fact one or more mirror-servers would be by far the best choice of backup method in this application, because in many cases record need to be deleted in such a way that it is impossible to recover a copy of that record.
Not that it would help in a situation where the records were erroneously but legitimately deleted - because they would be deleted from all the mirror-servers as well.
"Yesterday's article mentioned that there used to be a backup in the form of a complete mirror-server at a different location, but that was lost in a fire several years ago and so far not replaced."
<needlescratch/>
Wait, what?
In a former life I did some work which involved the PNC. One day in mid 2005 we lost all our connections. Turns out that the backup system we were using was in a building that was unfortunately close to a certain oil depot that had just explosively disassembled itself...
PLEASE tell me they haven't been running on a single system since then... !?
Mind you, having experienced PITO as a customer I wouldn't be surprised...
I have little doubt that Priti Patel has been hard at work and is spending many hours every day trying to decide who is the most convenient (and expendable) person to blame. Just like many of those whose records have been deleted, the question of guilt or innocence is completely irrelevant when compared with what is the most expedient.
I did note to my network admin (a couple of years ago, using win 7) that it was a bit unnerving that the rightclick 'delete' was perilously close to the 'rename' and 'create shortcut' function... don't worry that what the bin is for... a big problem is, if it is on a network share, it **does not** go to the bin, but disappears..
That was the original story I heard too, but both the volume and range have expanded since then.
There's been years of campaigning against police retention of data from those who are interviewed then released (or to use a technical term, "innocent"), or from mass 'round up the usual suspects' attacks on demonstrations. (Anyone remember kettling?) The police / government response was not much more sophisticated than "No. Shan't. Remember Ian Huntley?", despite him and that case being a red herring in this context. A lot of people will not be unhappy that those records have now been lost due to an admin SNAFU. Losing actual criminal records and those for investigations in progress is a little more serious.
What if this wasn't an accident? As everyone points out, this should be nearly impossible on a mature system like this, which should have backups and lots of barriers in front of operations such as this.
All you need to know is that an unnamed MP was recently released without charge for a (heinous) alleged crime. Perhaps there are other, unreported, cases as well relating to Tory MPs, or donors.
So this was potentially a planned data loss to remove the records relating to this, covered up with other deleted records, in a needle in a haystack method. They'll gloriously say they recovered 90% of the records in due course from backup or paper trail, but the rest of the paper trail for the critical records will have been disappeared. Did
Yeah, it seems far fetched and conspiracy theory like, and I'm not a fan of these in these times, and it most likely was poor backup practices and system access control. But still.
If I hear that somebody has been released without charge I assume it was because there was no provable case against them and that they are, therefore innocent. That is a basic principle of the English justice system (no "not proven" stuff here), has been for centuries and hopefully, despite the longings of the HO and intelligence services, will continue to be. It applies to everyone. It applies to Tory MPs and donors. It even applies to YOU. And if you think about it a little you'll realise that the reason that it applies to Tory MPs, donors and everyone else is so that it can apply to you. And if you do a little more hard thinking you might realise that that is the most valuable protection you can have under the law.
Meanwhile, those of us who've actually had the job of investigating allegations of criminal behaviour appreciate just how important it is that the subjects of those investigations do go unnamed.
This post has been deleted by its author
As everyone points out, this should be nearly impossible on a mature system like this, which should have backups and lots of barriers in front of operations such as this.
I think you are overlooking the minor detail of this being government IT, so hardly mature and apparently without backups. Besides that, those barriers also seem to have gone AWOL (assuming they ever existed).
Of course they do. They have an agenda. They've been for decades trying to make IT chartered profession where they can have out of date, expensive certifications.
Must've been over 2 decades ago when I looked and was like how much? For what? Sod that...
With hindsight, yes they are irrelevant.
> Nice of the BCS to blame the developers. This was probably user error compounded by worse management error because there were no backups.
Shirley, the BCS statement is making the point that it wasn't just the developer's fault because there were other people in the chain that should have checked as well but either didn't or also missed it?
AKA the Peter principle: people rise to the level of their own incompetence.
There's too many managers around that can't do the job of the people they're managing. Whilst there's *some* skilled and talented managers who understand this and effectively delegate and manage, alas they're a rare commodity outnumbered many times over by clueless management numpties throwing their (light) weight around.
"Quick, call the BCS" .. said nobody.
They're fine for academia (one ass-u-me s), but not when working at the coal face with unqualified 'managers' and dubious processes. Much like attempting to read 3 volumes of The Art of Computer Programming (Knuth) - which would probably qualify as membership, but you wouldn't get a job.
Yeah, a quick take-offline, restore and apply recent transactions from logs before re-onlining overnight and hoping nobody noticed the 2 hour outage. Been there. I deliberately mistype the initial parts of SQL statements now so they will fail in case I accidentally run them and then have an "Oh F**k" moment.
As a matter of course, I would generally write any DELETE statement as SELECT * first and check it returns the results I want.
Then, before running it, select the contents of the affected table(s) into backups, along the lines of SELECT INTO TABLENAME_20210119 * FROM TABLENAME.
Then, where practical to do so, start a transaction before running the DELETE and only commit it if I get the expected number of records reported. I say "where practical" because you might not always have the tempdb space available.
Of course, if you do actually want to zero out a table, you should be using TRUNCATE anyway, unless you have a particular need to be able to reverse the transaction log.
> Unless they are using the Adabas SQL gateway apparently. Adabas is not a RDBMS I'm familiar with, but Google is your friend (in this context).
When I were but a stripling starting out in the business there were job ads a plenty for Adabas and Natural. I nearly thought about re-training but never did. Eventually of course it was crushed by Oracle out-marketing everything even vaguely close as a competitor.
However it has made me suddenly realise that the poor person responsible is more likely to have made the mistake through senility rather than youthful exuberance!
When I were but a young'un, it was a case of get any job you can, and in my case, it turned out to be Equinox on Novell Netware (shudder). I was asked now to do anything with that now, I'm not sure that I could remember a single thing about it. And that's the way I want it to stay.
I had heard that the PNC has not been compliant with this legislation for years.
Scuttlebutt has it that it was the implementation of rules to become compliant which went wrong, and that this problem goes back to sometime in last year so some data will have rolled off their DR backups. If that was the case, then the rules to make sure that data that should be completely purged, even from the backups may have been incorrect.
Under UK and previously EU law, the police to not have catre blanche to keep information gathered from suspects forever. If someone is arrested for a crime, then the police can take and store information about the person. If they are subsequently released and the charges dropped, or go to trial and are cleared of the crime, then the police are bound by law to delete the fingerprints, DNA and other data that they've collected after a certain period of time.
Anybody convicted of a crime will have their records stored forever (incidentally, this is sometimes quoted as being why police will take fingerprints from people who are stopped for minor motor offenses, even when the data has no bearing on the crime - "it's standard procedure, sir"). Once someone has been convicted, the police have the right to keep any data they've collected whether it was needed or not.
This deletion policy does not please the police. They would really like to build up a complete database of all the people in the country whether they've been found guilty of a crime or not. If I were to put my conspiracy hat on (the one with the tin foil lining), this news story could be a deliberately created attempt to shock the people and government into a policy change to allow them to keep more information for longer.
Within the last decade, I heard a broadcast interview about the police's DNA retention policy with who ever was the chief of ACPO at the time, where he repeatedly called people who had been arrested but not yet tried as "criminals" instead of "suspects" (not even "potential criminals"), even after being pulled up by the interviewer on more than one occasion. I'm sure that some members of the police regard all of the public as criminals who just have not yet been caught yet!
In theory backups should be cleaned as well, but in practice that is usually not practical.The important thing here is putting the deleted data "beyond use". So it might still be on a tape/... but it cannot be accessed through normal use/mechanisms.
repeatedly called people who had been arrested but not yet tried as "criminals" instead of "suspects"
A crime is committed by one or more culprits and yet reports will almost inevitably say "suspect" instead. I suppose referring to suspects as "criminal" is the logical extension.
And the Libra magistrates court system that was memorably described as 'one of the worst IT projects ever seen' by the Commons Public Accounts Committee:
https://www.computerweekly.com/news/2240049341/Courts-Libra-system-is-one-of-the-worst-IT-projects-ever-seen
If they are able to recover records from backups then that sort of puts paid to the whole deleting-illegally-retained-personal-data stuff doesn't it? I mean, you sue the police to delete your records when you are cleared of a crime, order is granted but then even deleting the live record just removes the current version, no? Perhaps the background here is an effort to allow backups to be routinely restored in the future?
The RDMBS I used to work with included transaction log backups as part of the backup set. On a restore the transactions were restored as after the data. Consequently if the data backup were restored to the point in time of the last data backup, a point before a deletion took place, the subsequent restoration of the transaction log would roll forward the transactions so the deletion would be repeated.
The full backup set may physically include the deleted data but the restored backup doesn't. I regard this as the normal and satisfactory way to handle backups in relation to deletions.
This reminds me of a 'Goldwynism'. Legendary film producer Sam Goldwyn had the knack of saying or doing things in such a way, that they had people in fits e.g. "Include me out."..."A verbal agreement isn't worth the paper it's written on." etc etc.
When he asked his secretary what was in the 12 filing cabinets cluttering up the office, she told him that they contained every single detail of films going back decades, even to the start of silent films.
"Well get rid of them all," he said, "but keep a copy just in case."
Can't help but think what a gift this could be for all those pesky Freedom of Information requests....
Sir Humphrey Appleby:
Well, this is what we normally do in circumstnces like these.
James Hacker:
This file contains the complete set of papers, except for a number of secret documents, a few others which are part of still active files, some correspondence lost in the floods of 1967...
James Hacker:
Was 1967 a particularly bad winter?
Sir Humphrey Appleby:
No, a marvellous winter. We lost no end of embarrassing files.
Everyone mentions "failure in the backups" but doesn't cover other factors. This is a live system, presumably getting updates (new fingerprints, DNA results etc) on a minute by minute basis. Unless you've frozen update access, taken a backup, done your update and noticed any issues before unfreezing access, you CANNOT just go back to the backups without losing ALL changes made since the backup.
This was a "routine housekeeping data update" which could mean it wasn't done by the IT admins, but an end user data administrator. As such, there wouldn't be a change control round it to take backups and lock access etc, so there could be hours worth of data between backup and housekeeping.
Or, simply they didn't realise they'd lost the data until after their change window (which would be a failure in checkout). In either case, restoring the last backup isn't an option.
What's probably taking the time is restoring the data to a test system, unpicking the data they need to restore and figuring out how to add it back in while retaining data integrity in the production system. In most databases (particularly one which has "evolved" over many years with changing requirements), that's not going to be easy. Particularly when any subsequent failure will be on the front pages, adding to stress levels...
"What's probably taking the time is restoring the data to a test system"
And if they don't have a DR contract by which they've tested their restore finding that system and doing the restore are going to take quite a lot of time. That's before they manually start picking the records out. And that's before they graft the records back into the live data in a way that retains consistency.
Sure, this is a tricky process, but that's why there should be a prescribed procedure to follow. Nobody should be having to work out this shit in real time.
Still, somebody has to do it sometimes, I still have nightmares about the time I had to do it after a senior manager managed to delete the wrong half of a set of records in some related tables in the annual purge necessary to keep the numbers within the standard ranges. It took some time to fix everything, but as a result, some numbers in those tables were increased from six to eight digits, resulting in another set of all-nighters when some problems showed up only in production. Those were interesting times.
Yup. What was anticipated can have a pre-planned and possibly tested process working. What wasn't anticipated is where you earn your money.
Where what wasn't anticipate was the need to have somebody around to sort it out is where somebody gets to earn even more money.
>Everyone mentions "failure in the backups" but doesn't cover other factors. This is a live system, presumably getting updates (new fingerprints, DNA results etc) on a minute by minute basis. Unless you've frozen update access, taken a backup, done your update and noticed any issues before unfreezing access, you CANNOT just go back to the backups without losing ALL changes made since the backup.
That depends. If each record had a unique identifier/master key then restoring those records with unique identifiers that were missing wouldn't necessarily effect the addition of new records with new unique identifiers.
I was under the impression that the original DPA (before being overtaken by the GDPR) required that data be deleted, provided it was not too onerous to do so. Thus a person's record could be deleted from a live database and the hot standby, but allowed to remain on the backup tapes, because deleting a record from the middle of a tape archive was considered to be too demanding. Were records to be restored from the backup tape, the subject's record should not be restored, or deleted again if restored.
The same was true of Subject Access Requests and Freedom of Information Requests. The responder could claim the task of providing the information was too difficult/expensive to justify. For example, the Met Police in the UK are only now going to record the ethnicity of people they stop in cars, so previous requests to them to state the proportion of Black and People of Colour who were stopped compared to White people could be rejected on the grounds that 'we do not record that data':
https://www.theguardian.com/uk-news/2021/jan/18/met-police-to-start-recording-ethnicity-of-people-stopped-in-cars
This only after several cases of high profile Black people being stopped for apparently no good reason (MET discovers that 'Driving While Black' is shockingly NOT a crime).
Not strictly true.
For subject access requests is was the cost of providing a permanent copy. so for example if you SAR contained 20k pages of emails etc then you could argue it was too expensive to provide a copy BUT you would still be required to provide access. This could be done by providing access to thr information on the network using a dedicated dumb terminal. That's where Morgan Grenfell failed when attempting to refuse to respond to a SAR many years back.
In terms of cost FOI is quite simple its £450/£600 (18/24 hrs at £25ph) to identify, collate and prepare the data for the response. The fees are set by statute and you have to demonstrate your costs in order to apply S12.
With regards to the example of stop and search, although you are not required to create new data in order to reply to an FOI you are also expected to provide advice and assistance and if you know a) the no of people stopped and b) their ethnicity you would be expected to do the sums.
I know this is the case as FOI and DP is my day job and I have had to do the sums!!
Mines the one with the FOI Act with tabs stuck all over it in the pocket
I had thought that it was the Met who embarrassed themselves over stop-and-search, but Wikipedia claims it was City of London Police. In 2000 a middle-aged black man was stopped for "Driving While Black". Speaking afterwards, the gentleman concerned, Dr. John Sentamu (then Bishop of Stepney, and a member of the Stephen Lawrence Enquiry into Institutional Racism in the Police) claimed that it was the eighth time in eight years that he had been stopped in this way.
They’re definitely good at covering up.
But not that good ;0)
Perfect timing to break out the newly acquired laptop and plentiful backups to present a myriad of information to a new President with presentation of human rights abuses procured via psychological operations by his precarious predecessor and next chapter great gamesplaying for premium presentation :0) ....
Do they really use tapes anymore?
In a previous life [17 years ago] calls into the control room were recorded on tapes. This stuff needed replacing. We went out to the market stating that we wanted to use off the shelf hardware and record onto hard disks. The capacities of disks these days probably means that they can get a full years data on 1 hard disk [2 actually, one working disk and one backup]
Using the company software gave timelogs for anything needed for investigations and for instantaneous search and the ability to run searches of e.g. nuisance callers.
Off the shelf hardware makes upgrades costs effective.
What are the limits on RAID?
(Anon due to prior work in this area)
As part of the EU Exit work, a lot of data had to be deleted that they were no longer allowed to keep. This was mostly relating to European Arrest Warrants and other Schengen information (https://www.schengenvisainfo.com/news/uk-to-lose-access-to-million-pieces-of-information-on-criminals-in-case-of-a-no-deal-brexit/)
The timing of this 'loss' would seem to match up pretty closely with the timing of the 'hit go' instructions on that deletion.
To all the people asking 'what about the backups' when you remove this data, you have to remove the data from all the live backups, the offline archives, and even references to case IDs or other protected information that might be in log files or referenced elsewhere.
From the outside a guess might be that as part of this surgical removal of EU data, some UK data might have got mixed up, and the one-way delete-all-the-backups nature of the work has put us in this situation.
Finally the article mentions testing, I would assume that an operation of this delicacy would have been well tested, but remember it was only to be actually done if there was 'no deal' (or variations there of that didn't include access to this data, like the deal we finally got) and the last minute negotiations meant that there was no clarity on this until very late in the year, at which time there is very little time left to do proper testing before the date.
Clearly the obvious answer would have been to do a lot of testing for the different possible contingencies, starting a year or two ago, so that no matter the outcome we would have a production ready solution in place. Do you really think that is what was prioritised by management?
This is pretty much exactly what I have been saying, to which the responses have been along the lines of, "you must be some sort of Guardian reading remoaning snowflake libtard" (I paraphrase, it doesn't look like CJ has got here yet). I strongly suspect this is yet another of the amazing benefits that brexit is giving us, but some seem to enjoy keeping that cosy wool pulled over their eyes for that little bit longer.
Of course, we have no evidence that this has anything at all to do with the "b" word, but then again, reading between the lines, Priti Patel hasn't exactly been quick in coming forward with an adequate explanation, or indeed one that says, "this has nothing to do with brexit"...
I wonder if we will ever get a proper technical explanation of what has happened here. In theory, the purpose and workings of such a system should be public knowledge, even if the data held in it is not. The absence, or vagueness, of such an explanation in itself suggests something is being covered up, and it's not a step too far to conjecture that this is because it may be embarrassing to the government.
OK, but...
1. We didn't bomb out without a deal (nevertheless, the delete script may have been prepared)
2. What's in the 1200 page deal looks suspiciously like the EAW except it isn't called that.
Also, now we are out, we are even less beholden to EU's GDPR, so why delete anything?
...the loss of data during a “routine housekeeping” operation was used to justify removing the requirement for DNA profiles, fingerprints, etc of innocent individuals not connected to ongoing inquiries to be removed from police records.
No?
Me neither...
Yes the chain of evidence would have been interrupted. You would have to go back to the actual physical evidence which they would have had to have kept.
In fact, reintroducing 'dodgy' data brings into question the rest of the data on the system, how do you know what data was recovered and what was on there all along?
From the Beeb (https://www.bbc.co.uk/news/uk-55734593):
"The Home Office previously said that the faulty script was introduced in November 2020, but it did not run until earlier this month when the error within it immediately became apparent."
So, presumably no support/monitoring/control on the first batch run, inadequate testing, inadequate backout available after first run. How Not To Introduce A New Housekeeping Script 101...