Posts by Doctor Syntax 40557 publicly visible posts • joined 16 Jun 2014
"they are quite clearly and human stupidity error by the system architects. developers, Project Managers (who at least have teh excuse that they are just IT users and don't know better but really should) and Developer management who really should not this kind of thing happen."
More likely a human stupidity error then ensured professionals never got near it. Remember this is a government that declared its disdain for experts except on the rare occasion when it realised that it was in over its head.
The problem is trying to do things without getting techies involved. I doubt there's anyone in the techie side of IT - and even a good proportion of management who wouldn't look at what they were using and say "disaster in the making". In this case, of course, the management concerned did not come from that good proportion.
"The problem with CSV is that when you open it in Excel"
You don't bother opening it with Excel. You open it with an application which has been designed to check for integrity with such things as a declared number of data rows and a proper end of file marker. It fails? Then none of it gets imported and the sender is to to do it again, this time properly, and whatever management processes in place round it are left showing that data from that source is still pending.
It needs a proper process in place of which the file format are parts S/W and only parts. That needs proper management overseen by someone more capable than a two-time loser of PII.
If this biologist was in a testing lab with that job the first thing he would do to record results would be to fire up an RDBMS as he has been doing since about 1984.
However the problem here is to bring batches of results into a single system. In that case the first thing he would do would be to devise a file format with sufficient metadata (i.e. a row count) to check that the received file contained the intended complement of rows so that incomplete files could be instantly noticed. Very likely the format would be CSV. Only after doing that would the S/W be considered.
Taking time to do a little planning and design up-front pays dividends later.
The entire team responsible should be sent home to write out "Excel is not a database" by hand, 16,000 times. Each.
At the very least include a count of results uploaded at the top and check that that number is actually received. For preference also add a specific end of file marker. Ring alarms if one of these tests fails.
Is it really that hard?
"And yes, an excel spreadsheet is likely more secure than a database set up by someone who has no idea what they are doing."
And when the spreadsheet is set up by someone with no idea what they're doing you get the mess we're encountering now.
Step back here and ask a few simple questions (the answers might not be simple).
1. What data and in what format is received from the testing centre to accompany the samples?
2. What format do the lab instruments export?
3. What needs to be done to interpret the results from 2 and marry it up with 1.
4. How many systems need to be fed from this?
5. What format(s) are needed from that?
6. What needs to be done to prepare the output from 3 into 5.
7. What sort of scale is needed?
8. What's needed to accomplish 6 at 7 reliably and at appropriate speed?
9. What's needed to oversee the process and ensure its all running properly?
At that point you have a set of requirements for the processing needed. "A spreadsheet jockey" isn't likely to be the answer to 8 although it might e the answer to 9.
"Ok we should not use Excel. What should a non programmer use instead?"
Email or the telephone. To ask a programmer to put together a real application.
For a lot of purposes the spreadsheet is good enough but at the very least you need a sanity check on the results and sometimes "good enough" isn't really good enough and you have to require better. Dealing with people's health and lives is one of those times.
Unfortunately the costs of failure are still insufficiently high. Some of the breaches we've seen ought to have brought down the companies. It's not hard to think of a few who ought to be remembered only in MBA courses as case studies in failure. I can only think that C-suite members simply think "There, but for the grace of God, go I" and continue doing business with them.
"there was also additional challenges from businesses and people pouring random chemicals down drains"
We had a problem with the Belfast sink* (in Belfast) leaking. I had a few goes at tightening up the fixings with no success. Eventually the penny dropped. It was the sink we used to dispose of reagents used to prepare samples for pollen analysis. For soil samples this involved boiling with hydrofluoric acid.** It was an old building so the sewer pipes would also have been glazed; I wonder what it did to them.
* For those who don't know, glazed earthenware.
** Sporopollenin is amazingly resistant to all sorts of things.
"But one of those occupational hazards, hence why it's good practice to check a pit with a gas detector, not a lit roll-up ciggie"
It happened before my time but the scars were still there to be seen....
Gas leak in a tunnel, not sure what gas. Someone from the forensic lab went out to investigate with a detector. Only tested at one level and pronounced it clear. "Look I can strike a match." I've met the RUC sergeant who had the wit to say "Not until I get out".
Detached garage in the first house we bought in NI. Wiring in garage but with no direct connection to supply. The previous owners had managed this by taking a supply from the kitchen as and when needed with a long flex with a 13 Amp plug on each end. One end plugged into a socket in the kitchen and the other into a socket in the garage.
"Are we sure that the message wasn't just sent out by individual GP surgeries to all their own patients on behalf of the Government for England"
Yes we are. That's the whole point. It comes from "nhs.test.and.trace.covid19.app@notifications.service.gov.uk" which certainly isn't a GP. In my case, however, it's clear that the PII comes from my GP because it came to an address provided solely to my GP. It must, therefore, have been provided to them either from the GP practice or by the practice's data processors - of whom I think there are now two.
"perhaps the NHS provides a bulk email facility to which GPs can provide a mailing list"
It does indeed appear to come from a bulk email facility: notifications.service.gov.uk
It doesn't come from the GP. It doesn't have anything like "Envelope from" my GP. The actual ID there from which it comes is nhs.test.and.trace.covid19.app. A bit opaque, maybe, but possibly from the Serco business contracted to do the test and trace.
That service, in its short history, has form for email security: https://www.theguardian.com/business/2020/may/20/serco-accidentally-shares-contact-tracers-email-addresses-covid-19 and didn't see the need to refer themselves to the ICO over that incident. And in any event it's all under the control, for want of a better word, of Dido Harding who also has form with overseeing PII in her previous job.
Plenty of us commented here, right at the start, that trust was absolutely essential for us to have confidence in this operation and that HMGs of all colours have a long history of being untrustworthy in this regard. We also said that appointing Harding was a poor start to building that trust and this just confirms existing suspicions.
I don't suppose TT was trying to defraud their users. They just managed to lose control of the users' data twice to those who were trying. With Dido Harding presiding.
It now appears that T&T have acquired a large database of PII. With Dido Harding presiding.
T&T operation is subcontracted to Serco. One of their first actions was to lose control of their tracers' email addresses by email
I repeat again, the problem isn't the unexpected email, it's where the data has got to.
Tip: If your e-mail address actually contains the word "spam"
Sigh.
Some people just don't get it. Read this as many times as it takes to understand it: it is contrary to the DPA - based on GDPR - to pass on PII without specific, informed consent.
That is the problem here. Not the message on behalf of the NHS. The passing on of PII without consent. Who knows where it's got to once it's gone?
A similar issue:
Users were dropped into the Informix ISQL menu system at log-on end logged out when they quit the program. However the menu system allowed them to shell out by hitting '!'.
Solution? A quick program to mimic the menu system using the same sysmenu tables but without the shell-out option.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
