In wake of Horizon scandal, forensics prof says digital evidence is a minefield Digital forensics in the UK is in need of reform, says one expert, as the deadline to advise the government on computer evidence rules arrives. According to Peter Sommer, professor of digital forensics at Birmingham City University, various issues threaten the reliability of digital evidence, from the software used to gather …
There was a further complication in the Horizon case that, up until 2015, the Post Office was bringing private prosecutions (at least in England). The company was not only the source of the computer evidence but also the prosecutor - which you might consider to be at least procedurally imprudent. The whole system of private prosecutions needs reviewing as well as the rules regarding digital evidence.
Might I nominate HMRC who seem to be claimant/judge/jury and executioner all rolled into one.
Naturally they are infallible (which is why their latest power grab is an ability to help themselves directly from your bank account without further ado)
Oh and they will only speak to you on their terms (and provided they are not on a WFH jolly day)
The integrity of the data extracted is also pretty questionable when the data is:
a) stored by software clearly labelled as "beta", such as HMRC's own online Self-Assessment return software or
b) regularly corrupted by poor internal systems, such as those used for processing Real Time Information (for PAYE)
Rotten as the Post Office was and is, they were not acting as judge, jury and executioner but as investigator and prosecutor, as the police used to do in England pre-DPP and as, for example, the RSPCA still do now.
Judge and sometimes juries were involved.
Indeed. From the material I've seen about Horizon, it appears that:
The Post Office never issued even a Provisional System Requirement for Horizon, let alone a definitive System Requirements document.
In fact the material I've seen seems to indicate that the only system requirement definition was a single Post Office Sales Terminal implementation that was shown to Royal Mail executives at the beginning of the project. This was apparently never documented or used to generate system documentation for the central financial and stock control databases.
Similarly, no Acceptance Test scripts seem to have been written or used before Horizon was released for live operation.
In short, its difficult to see how anybody could have expected Horizon to have been bug-free, since almost every one of the generally accepted rules of computer systems design, documentation, implementation and testing seems to have been ignored.
Trouble is this situation is surprisingly common and has in many instances been normalised by the use: Agile, Scrum, Hackathons etc.
Aside: I think MS Windows 3 and Office 4.3 were the last versions of these products that had User Guides and Reference Manuals…
With financial systems this is even more of a problem because they do so much we take for granted, so the requirements are along the lines of: accounts, vat, HMRC and bank integration.
Has been required for a while.
Anything that can clarify how to validate digital forensic evidence, particularly for use in Court proceedings, will be a huge step forward.
If we can define a set of rules to define integrity of evidence, or at least some sort of playbook for this field, it will go a long way to presenting solid evidence in court.
Of course, the vary nature of technological complexities, will not make this an easy thing to achieve, but the fact it's being looked at is a step in the right direction.
Frankly for something as complicated as Horizon just merely snapshotting the data doesn't really get you very far. If you're just taking screenshots from a complex line of business application then we're in Alice in Wonderland territory because it can mean whatever I say it means.
Without requirements that describe what something is supposed to mean in context (and independent verification that what the requirements say is actually backed up by what the code actually does) I wouldn't be keen on expressing an opinion as a juror.
We have always known digital data is potentially flawed, sadly most of the public don't. Anyone working with IT would never have agreed computer data should always be regarded as reliable (as the Horizon case proved, software bugs meant the data was in places very unreliable, and as we all know bugs do occur. Cannot rely on most software inbuilt auditing either - I have worked with a variety of pieces of commercial software and in all those I investigated auditing could be disabled either via software itself if you had appropriate rights, or on underlying computer system, and beyond that default audit data retention was quite low - even saw some that had automatic date based purges that cleared financial data well before the 6/7 year rule (yes, people should be doing backups so audit data shouldn't be lost, but we all know instances of backups not being dore, or if they are the restore / retrieval process never being tested so faults not found until its too late) )
..though I would expect public to be well aware of bias in evidence that is presented, UK has a dubious record (in various police forces, not just a problem with 1 force) with hiding evidence that could at least give reasonable doubt or even exonerate defendant(s) (Birmingham 6 a classic example)
Horizon seems to have been a case of someone too close to the day-to-day operation becoming blind to its limitations, especially if there was incentive to be so. It takes a fresh pair of eyes to see the problems. That was how I conceived my role back in the day - a fresh pair of eyes. The police investigate, they bring in bags of potential exhibits, the statements, their hypothesis. Can I, without having been enrolled in any group-think that might have happened, find evidence that contradicts the hypothesis? If I can it avoids a miscarriage of justice, if I look hard and fail it strengthens the hypothesis but again it's up to the court to become the final arbiter of fact.
There were two sides, the gathering of evidence - in my day in conventional forensic science often delegated to police SOCOs - and the testing. I'm not sure this is brought out strongly enough; presentation and testing are not the same thing.
Horizon seems to have been a case of someone too close to the day-to-day operation becoming blind to its limitations
Or senior management (yes, I am looking at you Paula Vennells) deciding they really did not want to know what was actually going on, or that Fujitsu staff had direct remote access to systems in sub-Post Offices across the land and could edit records without hinderance or audit trail.
Digital evidence from the Horizon system was fraught with the problem of who actually input the data when Fujitsu denied remote access for so long, then admitted it. The real issue was that there was hard physical evidence that was never (AFAIK) even looked at. When the money the sub-Post Office has in its cash register at the end of the day does not tally with the amount Horizon says should be there you can do an old -fashioned stock take. How many postal orders really are in the drawer, and postage stamps etc etc. You can even set up an experiment to see what happens when all transactions are on camera and collate that with Horizon, or even check the paper audit trail printed out in the office for anomalies. None of this appears to have happened. That there is 'dereliction of duty' by someone IMHO.
Without a full audit, tracing individual transactions (a few billion by now) we will never know if money actually exited the PO due to people in the know helping themselves. However, we do know the end of year discrepancies were positive and simply got incorporated into profits and senior management bonuses.
Re: Post Office Horizon system landmark conclusion of the long-running scandal
I think you will find that this is still far from a conclusion. Project Horizon is still in use, Fujitsu are still expected to fix 'bugs' and many of the innocent Post office staff have yet to agree, let alone receive, compensation.
As I have understood the compensation procedure, where a claimant dies, the family/beneficiaries are entitled to the compensation payout, so waiting for more of them to die won't necessarily reduce the payouts.
I think it is more a case of keep kicking the can down the road in the hope that it willeither go away eventually, or that the potential compensation may be reduced (I mean by inflation rather than paying out a lower amount, although I don't think this would be allowed to happen either).
In this context, every book on digital forensics that has crossed my desk (as a reviewer or as study material) over the last couple of decades has concentrated on the technicalities of extracting data from devices, with little or no reference to actual forensics -- how to deliver evidence acceptable in a court of law. Even the admittedly pre-digital ACPO guidelines did better -- for example stressing documented chain of custody. And it's clear that, given Prof. Sommer's comments on idiosyncratic methods, this should be extended to include a clear description of any post-extraction processing performed (which should of course be made available to the court).
Emphasis on the word "forensics".
Present traditional forensic evidence to court, and the forensic officers (or lab staff) will be able to deliver - on demand - a list of the techniques they used, chemical & analytical processes (for DNA analysis, etc), show that the mass spectrometer was properly calibrated when you determined that the chip of paint came from the scene of the crime.
They have to do this because a lot of traditional forensics like DNA is effectively a blck box and courts have always said "well go on... how does it work if you're magically telling us this suspect was definitely at the scene of the crime".
As the article alludes to... "digital forensics" can be as rudimentary as screenshotting social media. Which is not necessarily a problem - scene of crime officers have long photographed rooms, scenes, etc.
But it needs to be regulated, with proper chain-of-custody and robust data gathering procedures in place.
I suspect the likes of Bellingcat1 have more robust audit trails than most Police forces (at least in terms of "casual" digital forensics conducted by non-specialist officers or detectives - not necessarily if they're shipping a device to a professional forensics provider).
1. Bellingcat have a home-brew audit package that basically logs the investigator's entire browsing history, shows exactly how they arrived at a page or document, archives it as they go - Recall-like - in the event a page or document is taken down or changed (for dynamic content), so you never get in a position where you say "can't find it again now" - particularly the case for social media where a timeline reload could whip away a post never to be seen again because the algorithm determines it's no longer of interest to you.
I was perhaps a bit loose in "traditional". "Biological" might have been a better word - fingerprints, chemistry, then blood and later DNA analysis - as opposed to "digital" forensics, which is more like traditional evidence-gathering - except the custody of a digital file can be a lot harder to document than a physical notebook you find in someone's pocket. That was perhaps the distinction I was making.
And the fact that whilst devices might be analysed by specialists, a lot of "digital forensics" might be done by non-specialists like detectives combing social media - whereas the same detective at a physical location might say "huh, there's a <thing> over there - can scene-of-crime bag it for evidence please and get it to the lab".
Just see the case of the "phantom of Heilbronn"
Investigators had connected her to six murders and an unsolved death based on DNA traces found at the scene, plus traces they found at another 40 crime scenes across southern Germany and Austria.
It was determined that the cotton swabs used to collect DNA had been contaminated accidently by a woman working at an unidentified factory in Bavaria. One company making swabs said they were not intended for analytical, but only medical use, while another said that there had been no requirement for the swabs to be free of DNA.
None of this seems very different from the issue of trust and reliability in traditional forms of evidence such as witness statements, interviews, photographs etc. Even without the issue of deliberately altered or faked material.
It seems like we just need to move ourselves away from the "the computer is always right" mentality. I'm happy with "the computer is always precise"* but always remember that precision and accuracy are different things.
*Unless using a quantum computer of course.
If one were to describe a range of digital device forensic range techniques, surely the use of trace evidence from a digital device would be at one end, and the manner in which "digital data" was used in the Horizon cases the other?
To use a car analogy, pulling timestamped throttle position and braking activation information from an ECU and using it to infer what actions a driver was taking would be very different from telling a jury that they must assume a Tesla Full Self-Driving implementation was operating correctly when determining the cause of an automotive crash.
From what I have read about the Horizon debacle, it wasn't so much that people were told that digital data found as trace evidence must be assumed to be correct (such as evidence from logs or deleted files being used to show user activity), it was that people were told that the functioning of a complex financial application with a multitude of privileged administrators must be assumed to correctly show the behavior of unprivileged end users. For the former, I expect that one would have to show that the operating system was likely to be functioning correctly and that it was unlikely that a third-party's actions could also explain the evidence. For the latter, I would expect that one would have to show that the complex financial application functioned correctly and that it was unlikely that a third-party's actions could also explain the evidence. The failure in the Horizon case seems more complete ignorance of forensic science in the digital domain rather than a poor application of forensic science in that domain.
If you want something like logging to be secure having the logs be placed on a blockchain at the time the data is logged makes them resistant to later change or deletion. That's not appropriate for all types of digital evidence, and you can't do it retroactively, but if e.g. you log the digital signature of emails including the time of receipt on a blockchain then you could prove that an email used in evidence is the same one that was received and the time it was received.
I'm completely out of touch with the latest iterations of blockchain technology, but isn't there some issue with exponential increase in computation as you add to the chain? If so, tends to make its use for log/audit immutability proof a tad tricky.
Not for this the specific use case.
I would use the term "Merkle tree" rather than "blockchain" here because what the parent poster is thinking of is tamper-evident logging.
When you use a Merkle tree to make a tamper-evident log, each entry in the log message contains a signed hash of the previous entry. This is fast. Also you don't use proof of work, which is where the never ending increasing power waste in bitcoin comes from. Instead you have multiple organisations (that you trust to not all collude with each other) sign log entries regularly. Certificate transparency logs work roughly this way.
Tamper evident logging is IMHO a perfectly good and reasonable idea for use in establishing evidence of chains of custody, though hardly a complete solution by itself.
digital forensics is hard but if there was an automatic custodial sentence for boards who didnt keep proper records, can guarantee it would then get done
neccessity is the mother of invention
cant see it changing as the companies/consultants are making monies
and everyone from politicians to corporate types happy to live behind the "tech error" when they were about to get caught
ill eat my shorts if any exec in the horizon scandel ends up going to prison... thats for the plebs who cant afford the highly paid (morally difficent) clever legal big wigs
its all sophistry jim .... but not as we know it, as its techy wechy :)
IMEI based evidence too.
I've heard of cases where despite damning 'evidence' of someone's absence at a crime scene they were still convicted, because one critical detail was not mentioned to the defence and prosecution - that an IMEI / IMSI can be spoofed. Once this happens, literally anything else is possible from retransmission of packets to another cell (doable) to replay attacks when an individual is halfway across the country. Any discrepancies can simply be explained away as 'Obfuscation' by the defendant or their accomplices or just plain old network noise.
Same with a drive, it is only too easy to install malware that creates a convincing looking porn site-porn site-email-Facebook-etc digital footprint using cloned hardware then plant the 'Evidence' on someone's laptop or other device via any number of exploits and zerodays.
Even temporary files can be copied over, producing convincing evidence of guilt despite someone having never visited a given site or sites.
This is sometimes done in messy divorce cases as 'Evidence' to convince the lesser lawyers on balance of probabilities that someone has been cheating, when in fact they have not been to convince the family courts that someone is doing something they shouldn't be.
Misconceptions:
POL did the prosecution so not impartial. Relying on the incompetence in the CPS and METS would have resulted in the same outcome. With only one redeeming feature that it would have taken longer or the case would be dropped due to other priorities or cost (POL had a blank cheque from the taxpayer).
Evidence was flawed due to buggy Horizon. All computer systems with their many third party components and hardware will be buggy. By this definition digital forensic will always create doubt.
Evidence was unreliable due to (uncontrolled) remote access. All computer systems require some maintenance functions. How well controlled will then always create doubt
Evidence was not independently verified. Well, we had Gareth Jenkins who claimed everything in so far as the evidential data for the charges was correct. Not an independent expert but not likely to find anyone else with sufficient knowledge. Even someone brought in by the defence could not see any holes in GJ evidence. Bottom line, such digital evidence can be manipulated both ways in reality.
Horizon not fully specified or tested. How many such systems can really be specified and be fully tested. Much will be requirement creep (through the complete life-cycle) and proper verification (plus re-verification) will be during real use. So digital evidence will always be not entirely useful evidence.
Digital forensics is a field that is generally a bit imbalanced in terms of the minds available. It's lopsided in favour of academics who have very little real world outside street wisdom.
A mate of mine dragged me into a forensic investigation once that he was doing on behalf of some lawyers in a tax lawsuit on the prosecuting side, might have been HMRC, I don't know I wasn't really privy to much. I was brought in because they were struggling to access things like email etc because they didn't have passwords or anything, he was dragged in because the previous guys couldn't get anywhere and they thought they needed cybersecurity expertise...they had full images of everything and had indexed it in a forensic analysis tool...Autopsy, from the outside it all looked very professional...the data had been through at least 5-6 investigators before it got to me (integrity was intact because everything was read only etc etc proper procedure had been followed and so on, you know...the academic stuff)...anyway, the previous guys were so caught up in process and using fancy tools and so on that not one of them had searched the indexed data for the word "password"...I've worked with enough guys in the same profession as the perp to know a common pattern amongst them, so I did and within 2 minutes I had a spreadsheet with all the passwords in, perp wasn't smart at all he was bog standard in the way he operated...from there it was less than two hours to find the information they needed...this was after 3-4 months of other guys working on it, doing deep searches for very specific strings, trying to figure out if there were hidden encrypted volumes all of that bollocks, you know assuming that the guy they were investigating was some kind of world class master of espionage...they were at the point where they were considering another warrant to seize more hardware from more extended family members and such, it was a parody.
I think the main problem with digital forensics is that it is too forensic, too academic. It's very easy for the process to be too complicated and focused on itself rather than the goal at hand leading to the basics being missed.
Over collection of hardware / data is also problematic. It is very tempting to just "take everything in case we miss something" but in the case of digital forensics, if you do this you end up searching for an eyelash in a skip full of pubes and that can send you in wild and whacky directions making dumb assumptions...it's also an indicator that you probably don't know what you're looking for and if you did, you don't know how to find it.
The sheer stack of seized kit was insane and far beyond what was required. It went as far as DVD players from the attic and the kids tablets. Bin bags full of rubbish, empty suitcases from the attic...fucking loads of pointless shit.
Sure, I get it...the perp could be hiding shit anywhere...but the possibility of something doesn't make it probable. In most cases, people just generally aren't that smart. At least not in the cases I've been involved...most crims operate under the assumption they won't get caught so they don't take opsec seriously or they just don't apply any...I mean the dead giveaway for me was this his laptop was smashed up...someone with good opsec doesn't have to smash their laptop up...know what I mean?
Anyway...data forensics, not unlike cybersecurity, has become a field largely dominated by academics and folks with absolutely no background in anything other than their own field and very little human interaction outside of academia and they're hired by the same types of people this leads to questionable sweeping decisions that have massive knock on effects further down the line which creates very complex situations...in the case above simply trying to find an email password turned into a hunt for veracrypt volumes and other wild shit because they couldn't see things from the perspective of the person they were investigating, they could only see things from the perspective of an ever escalating theoretical academic thought experiment.
As someone who is in IT and Finance - Horizon should never have been able to adjust ledgers/journals in such a way to
a) avoid any audit traceability - not even sure that's following any sort of financial audit standards.
b) correct accounting - should have had multiple accounts and double-entry into ledgers to show where/from any changes to accounts were made and by whom.
An important aspec that is widely overlooked is the role of the judiciary / courts in all this.
Down the decades, the courts have been repeatedly "hoodwinked" by suppliers of specialist evidence. There's many instances, e.g. Roy Meadow, the highly respect pediatrician who had a habbit of dishing up "evidence" to courts resulting in many women - recently bereaved after their children died of cot death - were jailed. In the earlier days of DNA evidence being used, the standard for "matches" was too weak, resulting in people being wrongfully convicted; that got corrected only when a wealthy convict was able to demonstrate that, whilst he was a close match to the evidence, he was not in fact an exact match (and therefore clearly not the person who had carried out the crime). There's been this post office scandal, where the courts have allowed an evidence *type* of a very dubious nature to be used *700* plus times. In Scotland there was the scandal about fingerprint evidence being made up. There's been cases in the Family court where social services armed with bogus "evidence" have repeatedly pursued families (there's one case in particular that hit the Sunday Times, and was resolved only when a French court granted the family asylum and protection from the British courts / social services). There's been many more.
Not once in any of these judicial failings has the courts intervened, and asked "hang on a mo, are you sure?". Leaving it to the prosecution and defence to "argue" (to the extent they're permitted) about technical or scientific is lunacy, and the courts do not even all the argument about such evidence to be fully explored (they know juries cannot follow it).
The most important thing that should come out of this is that the courts / judiciary themselves need to wise up on science and technology, and become more of an "inquiry" rather than an "argument". I think this is how things are done on the Continent, with courts being inquisitorial rather than adversarial. It seems to me that whilst our courts are adversarial, we're going to continue to have screw ups like this.
The fact that the courts have repeatedly been hoodwinked and that the courts / judiciary have not acted to prevent that happening again suggests to me that they're as culpable in the Horizon Scandal, and every other similar judicial scandal. If they continue to refuse to acknowledge that they have a problem, they are not doing their job.
Firstly, I overall agree with the points that Prof. Sommer raised. Although I would point out the way law enforcement has gone, is following the ISO standard. Whilst I can see this has a value, it's led to examiners that follow a set process without thought for what they are doing. Then due to the sheer volume of work versus staffing levels, the emphasis is put on throughput of work and not eye for detail, with the idea being that, 'if it gets challenged, we'll spend more time on it later'.
Over the last 20+ years I've seen a stream of examiners who got caught in the 'CSI effect'. Where they think it looks 'cool' on TV and they want to do it. The universities then provided, expensive but mainly, pretty useless courses on the subject to follow the trend (and money). It is also worth pointing out the number of university places versus the number of available job positions don't tally. Don't get me wrong, some can be trained and make good examiners and some are great at following the set procedure in front of them but can't do anything else if an issue arises.
The software was frequently updated, but as highlighted, this wasn't always a good thing and often led to spurious results. I've had to revisit many of previous cases after an 'issue' has been discovered in the software to see if that issue would play a part in my case.
Luckily for me, I found an escape from the law enforcement side of forensics, as sadly, it's not what it was or anything like it should be, to the point where I genuinely feel that if I was so inclined to perform defence work, I could find enough fault in the vast majority of computer and phone examinations to bring the evidence into question.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
