Posts by David Harper 1 284 publicly visible posts • joined 12 Jun 2009
I also worked with some of the original Cluster scientists back in the 1990s, and I remember the deep despondency following the failed Ariane 5 launch. Many of the early-career scientists were particularly devastated. It's a testament to ESA that the mission was re-built in under five years, and that the spacecraft continued operating for more than ten times the original mission lifespan. What NASA can do, so can ESA.
There's already a digital ID system in the UK. It's called the e-Visa, and all non-British residents must now have one. Last year, it replaced physical proofs of right of residency such as indefinite leave to remain stamps in passports. And it has been a disaster, with many e-Visa holders finding that the app fails to work at critical moments such as when they are trying to board a flight back home to the UK. Other e-Visa holders have found their details being mixed up with those of other people by the Home Office's computer systems. In short, a typical government IT fiasco, but with the worst aspects of the Windrush scandal thrown in for good measure.
My American wife, who had indefinite leave to remain in the UK for almost 30 years, decided to acquire British citizenship rather than risk being stranded overseas by the e-Visa system. Perversely, it was also easier to do that than to convert her ILR into an e-Visa.
I've followed the ongoing e-Visa fiasco for a couple of years now, and the idea of being forced to carry something similar horrifies me. If cancelling the digital ID project is the only thing the Burnham government achieves, it will still be a good thing.
I've been using the command line for almost fifty years, on mainframes, VAXen, various flavours of UNIX, and latterly Linux. I'm old enough to have used systems where the documentation consisted of an entire bookshelf of enormous printed ring-bound volumes, with an index volume to tell you which of the other volumes you had to read to find the options to the command you wanted to use. By comparison, running "man <command>" is a walk in the park. But not, apparently, for some lazy buggers. Time to retire, methinks.
"It is, of course, getting worse. The whole idea of agentic AI is pinned to the donkey by the assumption that your agents need your access rights to act on your behalf. There being no industry-wide best practices, no inherent management principles, or indeed inherent anything, this means giving AI agents your passwords – something that in a sane and godly world you would not do."
No industry-wide best practices? You mean, like OAUTH, which has been around for two decades?
My tale is from the early 1990s, before PowerPoint was invented to bring joy into our lives, and presentations were done using acetate sheets on an overhead projector (kids, go ask your grandparents).
Our department's very expensive photocopier was rendered unusable when someone fed regular acetate sheets through it instead of the special heat-resistant ones. The sheets simply melted onto the drum. The perpetrator had fled the scene, but he had left the originals of his slides in the machine, so the departmental administrator was able to identify him and deliver an appropriate bollocking.
About 20 years ago, the Open University ran an excellent course called "Learning from Information System Failures" as part of its postgraduate computing programme. It examined the different ways in which large IT projects often fail. The course material included several case studies, including the infamous Cambridge University CAPSA project and the Home Office's 1999 Passport Office fiasco. If the OU ever brings back this course, I predict that the Digital ID project will be a prime candidate to become a new case study.
It looks like Ray committed a file containing production database login credentials into the company's source code repository. That is shoddy practice on so many levels. If Ray had worked in my organisation, he'd be getting a very severe bollocking from our cybersecurity team *and* our DBA team.
The Home Office is already trying, and failing, to implement a digital ID system for millions of foreign permanent residents living in the UK. It's called the eVisa system, and it is riddled with flaws. The campaign group the3million has recently published a detailed report on the failings of the eVisa system (https://the3million.org.uk/digital-status-crisis) which should be required reading for anyone advocating for a nationwide digital ID system.
On the plus side, given the track record of large, complex government IT projects, this one is never likely to get off the ground.
All major relational database systems allow DBAs to control what each user account can do by granting different levels of privilege, all the way from superuser status down to "you can connect to the database but you can't see or alter any data". So why -- especially after the first incident -- didn't the DBAs severely restrict the reporting account to allow it only to execute SELECT? That's what we do in my organisation. If you have a valid need to view data, but not to alter it, we grant you SELECT privileges only. You can try to execute INSERT or UPDATE or DELETE, but the database will respond with "you don't have the privileges to do that".
It's called an eVisa, and the Home Office has been issuing them to foreign students, workers and non-British partners of British citizens since last year. Existing non-British residents with physical immigration documents, such as stamps in passports or biometric residence permits, are now required to exchange them for an eVisa.
Employers and landlords are legally required to verify the eVisa status of anyone they want to hire or rent to. It is also checked by airline staff before boarding a flight to the U.K.
The Open Rights Group published a report last year (https://www.openrightsgroup.org/press-releases/broken-e-visa-scheme-could-lead-to-digital-windrush-scandal/) highlighting serious problems with the Home Office's implementation of the system, and warned that it could lead to another Windrush scandal. Readers of The Register will find the ORG's report both fascinating and alarming, as an example of a major government IT project that has fatal flaws.
Wilma was extremely lucky that there were backups of the other web sites, but her first mistake was to begin by DELETING the old web site directory, when a more experienced person would have RENAMED the directory e.g. "mv website website.old", before installing the new web site alongside it. That way, if the new version didn't actually work in the production environment, the old web site could be restored in seconds with a couple of directory renames.
Still, the wise learn by their mistakes.
"We may live in a galaxy teeming with life, but very little multicellular or intelligent life."
As the Monty Python team once memorably expressed it:
So remember when you're feeling very small and insecure,
How amazingly unlikely is your birth,
And pray that there's intelligent life somewhere up in space,
'Cause there's bugger all down here on earth.
Bu those legacy systems also contain undocumented workflows and business logic due to (in the words of the article) 'encode[d] arcane policy'. You could be a COBOL grandmaster, and still not be able to figure out what the f**k the program is doing. Anyone who has ever been told to debug someone else's 10,000-line program will know what I'm talking about.
Considering the billions of pounds that have been wasted trying to integrate NHS medical records over the years so that Hospital A can view your records from Hospital B or GP Surgery C, it's ironic that at least one hospital in the U.S. appears to have managed it quite by accident.
Twenty years ago, the Open University offered an excellent six-month graduate-level course called "Learning from Information System Failures". The course materials included several case studies of large IT projects that had failed. One was the disastrous Cambridge University Capsa project of the late 1990s, which attempted to implement a single accounting system across all the colleges and departments of the university. If the OU ever revives the course, Birmingham City Council's Oracle ERP project will surely be a prime case study.
As a Ph.D. student at Liverpool University in the mid 1980s, I tried to use the X.25 JANET FTP protocol to transfer a set of data files from a VAX at the Royal Greenwich Observstory (of blessed and glorious memory) at Herstmonceux to the IBM mainframe at Liverpool. It turned out to be a non-trivial task, and even when the head of operations at Liverpool got involved, he was unable to get it to work, and advised me to ask my collaborators at the RGO to put the files on tape and send it by Royal Mail.
There are Babylonian clay tablets containing cuneiform text that are almost 4,000 years old and still readable. Some of them contain records of solar eclipses and other astronomical events that are still being used by modern astronomers to investigate how the Earth's rotation rate has changed over the millennia. Now THAT is durable data storage!
If I were cynical, I might wonder whether this is simply a case of AI being used to generate greater volumes of the kind of corporate BS that blights the lives of the people who do the real work. I think readers of El Reg will know what I mean: emails, reports and presentations from HR, senior management and consultants, full of buzzwords but oddly devoid of any actual meaning. On the plus side, if I can toss these into Copilot and ask it for a one-line summary in language that a 7-year-old can understand, I'll save myself actually having to read all of the tedious crap and save myself a lot more than 26 minutes each day.
Ah, you're talking about MyISAM, the bane of every MySQL DBA's life. Yes, legacy applications that rely on MyISAM are a world-class PITA. I used to work with a team that refused to switch to InnoDB because they liked to move entire schemas between instances using filesystem-level copying of the data files. You could (with care!) do that with MyISAM. The manual says that you can now do it with InnoDB using the "Transportable Tablespace" feature, but any user who comes to me and asks about that will get the Paddington Bear Hard Stare.
"MySQL broke this by supporting, but ignoring the syntax, and using table scans whenever tables were joined, causing many developers to think, not unreasonably, that JOINs were the problem."
I've read and re-read this several times, but I still don't understand what you're saying. MySQL has always used indexes to perform joins, if there are suitable indexes. Its query analyser may not always pick the best index for the job, but it does have the EXPLAIN command, and an experienced developer or DBA will always run a new JOIN query through EXPLAIN to find out what indexes the query analyser is planning to use. MySQL also has a handy extension called index hints, which allow you to say to MySQL "no, don't use that index, use this index instead", when EXPLAIN shows you that it's picking the wrong index itself.
Occasionally, the query analyser will determine that none of the available indexes will give better performance than a full table scan. That can happen if you have an index on a column that has only a couple of values, and the data are split 50:50 between them. That index is pretty useless, and a full table scan will in fact be faster. That's not MySQL's fault, of course, but trying to explain that to some developers is like trying to explain quantum mechanics to my cat. Pointless and annoying for both of us.
"It is a long time ago but I remember having to fix a website where a single user had two different ids… and the root cause was the lack of data integrity across tables that a FOREIGN KEY would have ensured."
I've seen this myself, back when Ruby-on-Rails was (briefly) the must-have framework that all the cool kids were using. I was told, by someone whose job title was "senior developer", that they had no need for foreign key constraints, because ActiveRecord (RoR's ORM layer) took care of data integrity. A couple of months later, the team leader asked me to look at the possibility of adding foreign keys. If you know MySQL, then you'll know that it won't allow you to add a foreign key if it would be violated by existing data. That was the case with several of the critical tables, so I had to tell the team leader that the data were already inconsistent and would need to be cleaned up first. Thankfully, that wasn't my problem.
And that's why, whenever I hear a developer say "MySQL has changed my data!", I roll my eyes.
Well, the developers *would* say that, wouldn't they. After all, every developer will tell you that their code is perfect, and if weird sh*t happens, it must be the database, right? :-)
Seriously, if there were a problem like the one you describe -- MySQL randomly changing data -- then support forums like Stack Overflow would be full of threads on this subject, and it would be front-page news here at The Register. Googling for reports of MySQL changing users' data unexpectedly, I found just one Stack Overflow thread (https://stackoverflow.com/questions/49594025/mysql-loss-of-data) where a user claimed that MySQL was changing their data, and the leading reply was as skeptical of the claim as I am.
Occam's Razor says that the simplest explanation is the most likely: the bug is in the application, not the database.
"PostgreSQL's backup facilities are to say the least primitive"
That may have been true ten years ago, but the pg_basebackup tool has been part of the standard PostgreSQL distribution for more than a decade now. It allows you to make hot backups of both local and remote PostgreSQL clusters, and re-building a working database cluster from the resulting backup fileset is so easy that even an intern could be trusted not to screw it up. The capability to backup a remote database cluster also makes pg_basebackup the perfect tool for setting up standby clusters quickly and easily. I'd hardly characterise those kinds of capabilities as "primitive".
"I am speaking of experience with a database holding 3 million entities (a song metadata database). It would always lose records and never knew the exact number of songs."
Is it possible, do you suppose, that your software developers weren't very good at their jobs, and the reason why the records went missing is because the application was defective?
I ask because I've worked with MySQL for 25 years, as an application developer and as a DBA, supporting databases with tables holding hundreds of millions of rows, and I've never seen the kind of behaviour that you describe.
