back to article What a Hancock-up: Excel spreadsheet blunder blamed after England under-reports 16,000 COVID-19 cases

As the UK heads into a troubling second wave of coronavirus cases, those in contact with thousands of people who just tested positive for COVID-19 in England went about their lives for up to a week unaware they had rubbed shoulders with a carrier. It's estimated as many as 48,000 people were not informed they had come in close …

  1. Mike 137 Silver badge

    'spreadsheet software as "human middleware" in the sector'

    Spreadsheets are "human middleware" almost everywhere, not only in the NHS. I've only a couple of times in as many decades seen a "risk register" that wasn't an Excel spreadsheet. Indeed the limitations of a spreadsheets continues to inhibit proper risk assessment, as it has to be hierarchical from scenario to causality in order to arrive at realistic results but nobody seems to have found out how to do that in a spreadsheet.

    And although it's not so disastrous, I've also seen spreadsheets widely used a layout grids for corporate tasks that have no requirement for calculation, such a organisation charts and customer enquiry forms.

    1. Arthur the cat Silver badge

      Re: 'spreadsheet software as "human middleware" in the sector'

      Spreadsheets are "human middleware" almost everywhere, not only in the NHS.

      My other half does a lot of consultancy work for various HMG departments. They usually want spreadsheets as their deliverables because a) the politicians and civil servants know Excel and don't want to learn anything else, and b) Excel runs pretty much on every departmental computer so the spreadsheets will run(*) anywhere. The downside is of course that bugs in spreadsheets are ubiquitous and a bastard to find.

      (*) Some are so huge that "crawl" would be a better verb.

      1. Doctor Syntax Silver badge

        Re: 'spreadsheet software as "human middleware" in the sector'

        Good enough for politicians but real work needs something better.

        1. keithpeter Silver badge
          Childcatcher

          Re: 'spreadsheet software as "human middleware" in the sector'

          The humble spreadsheet is the last bastion of end user programming and as such I (a clueless end user) shall defend it to the hilt. People (aka stakeholders) can puggle round and sort their logic out and (one hopes) validate against results in a small scale trial. That obviously did not happen in this case.

          If there is a need to scale up and/or if the application becomes critical, at that point they should be handing over to software professionals. The use of an Excel file type in an automated data acquisition chain is a really bad smell generally. That is the issue I think.

          https://www.wired.com/2014/10/a-spreadsheet-way-of-knowledge/

          https://baselinescenario.com/2013/02/09/the-importance-of-excel/

          https://www.forbes.com/sites/timworstall/2013/02/13/microsofts-excel-might-be-the-most-dangerous-software-on-the-planet/#557ad19272ae

      2. anothercynic Silver badge

        Re: 'spreadsheet software as "human middleware" in the sector'

        This is true... a friend of mine with extensive experience with the Cabinet Office and other departments made precisely that observation earlier today...

        Excel is wedged into places and made to do things it's not designed for, but because that's all people care to understand, that becomes the default format. It's truly horrific.

        1. gobaskof

          Re: 'spreadsheet software as "human middleware" in the sector'

          It is true in so many places. You even see some universities teaching Physics students to do complex data analysis in it simply because some old incompetent fart that teaches the course has never learned to program.

          It has now got to the point where if I see a spreadsheet used at all I have a instant angry reaction. Occasionally I then have to apologise when I see that what is being done are simple accounts, something that a spreadsheet was actually designed for.

          1. This post has been deleted by its author

            1. Arthur the cat Silver badge
              Alert

              Re: 'spreadsheet software as "human middleware" in the sector'

              Nightmare concept: a spreadsheet which uses Fortran IV for its formulas and macros.

              1. plrndl
                Joke

                Re: 'spreadsheet software as "human middleware" in the sector'

                WHAT'S WRONG WITH FORTRAN IV?

              2. GreyWolf

                Re: 'spreadsheet written in FORTRAN IV

                Hey!

                I got a prize for a spreadsheet written in FORTRAN IV! Dinner for two voucher at any good restaurant.

                Mind you, I had no idea that it was generalisable, never heard of spreadsheets...slow on the uptake, that's me.

            2. GreyWolf

              Re: Old Fart

              I am an old fart, marginally competent, and I learned to program in FORTRAN II (that's 2, to you young uns who never mastered Roman numerals).

              I too hate Excel and anyone who thinks it can be used for serious work.

        2. Doctor Syntax Silver badge

          Re: 'spreadsheet software as "human middleware" in the sector'

          Is it too much to hope that fiascos like this lead to the recognition that it has its limits?

          Yes, it almost certainly is.

        3. dajames

          Re: 'spreadsheet software as "human middleware" in the sector'

          Excel is wedged into places and made to do things it's not designed for, but because that's all people care to understand, that becomes the default format. It's truly horrific.

          Yup.

          Some years ago I was writing software for a manufacturer of security hardware. Whenever we added functionality to the boxes we had to generate and document some test cases that could be included in the company's test regime (such as it was). This was before the days of continuous build/test/release cycles and everything was manually initiated, though as automated as we could make it.

          Generating the test data was OK, but writing up the tests and their intended results was ... tedious.

          So ... having got bored with typing up test data the hard way I wrote some VBA that called an encryption library I'd written and built as a DLL. The VBA wrapper converted between binary strings and readable hex and pulled values in and out of cells in an Excel spreadsheet (16-bit Excel, at that) and the DLL did the actual crypto calculations.

          Whenever someone said "We'd like to run the same test as this, but using a different transport key" (or whatever) I changed the data in the spreadsheet and printed it out (yes, on paper, it was a while ago).

          Excel was NOT meant to be used like that ... but it was meant to be extensible and some of the things that people have done with it are pretty impressive. Just think of it as a rather over-featured grid-control.

          Just keep it out of the hands of politicians!

      3. veti Silver badge

        Re: 'spreadsheet software as "human middleware" in the sector'

        Spreadsheets are fine for *presenting* data, and OK for "analysis" provided your analysis doesn't require adding columns. It's when people use them for *storing* data that the rot sets in.

        1. Sanguma

          Re: 'spreadsheet software as "human middleware" in the sector'

          It's when people use them for *storing* data that the rot sets in.

          Fair curls my toes. You lose the advantage of ACID (atomicity, consistency, isolation, durability) that comes standard with most reputable RDBMs and instead get a pile of arithmetic operators you can't use with alphabetic characters.

          The irony is that even Microsoft put something in Excel to communicate with RDBMs, or at least with Access, and that's better than nothing. But nobody's thought of using it.

          1. Captain Scarlet
            Mushroom

            Re: 'spreadsheet software as "human middleware" in the sector'

            Except when they do use it they do silly things, one most often I see is using Having instead of from statements.

        2. Rol

          Re: 'spreadsheet software as "human middleware" in the sector'

          Thankfully, Microsoft has been working very hard to discourage the use of Excel beyond adding up a child's pocket money allowance. With every new release they add in more unfixable "improvements" that aims to end the use of Excel in the office for good.

      4. Charlie Clark Silver badge

        Re: 'spreadsheet software as "human middleware" in the sector'

        Excel isn't too bad* as a report format because it is so ubiquituous and doesn't suffer from some of the many problems associated with CSV, which is okay an an intermediate format. It's also better in many situations than PDF. But using it for data processing is asking for trouble.

        * As someone who maintains a library for processing XLSX files I would say that, wouldn't I? ;-)

      5. Anonymous Coward
        Anonymous Coward

        Re: 'spreadsheet software as "human middleware" in the sector'

        Of course not only in the NHS. Track and Trace is Serco, a private leecher of public money. PHE while a government agency is also not part of the NHS.

    2. John Smith 19 Gold badge
      Unhappy

      Hmm. 65 000 000 people. 1 000 000 col limit*

      Who could have seen this coming?

      *And that's best case for the new format.

      1. JDX Gold badge

        Re: Hmm. 65 000 000 people. 1 000 000 col limit*

        Unless everyone gets tested on the same day this is probably not an issue.

        1. Eclectic Man Silver badge

          Re: Hmm. 65 000 000 people. 1 000 000 col limit*

          But HMG wants to test the whole population in one week, eventually. So over 8 million per day (I am excluding babies here).

          This is not a car crash, this is a multiple pile-up.

          1. heyrick Silver badge
            Mushroom

            Re: Hmm. 65 000 000 people. 1 000 000 col limit*

            "This is not a car crash, this is a multiple pile-up."

            To bugger up tracing people during a pandemic, this isn't a car crash. This is a plane crash into a train crash into a multiple car pile-up with a tanker of petrol on its side and currently on fire and...... (see icon)

          2. JDX Gold badge

            Re: Hmm. 65 000 000 people. 1 000 000 col limit*

            "eventually" - so new tools will be discovered on the IT side. Just as the original test infrastructure had to be totally reinvented (maybe more than once) to get to 5, then 6, then maybe 7 figures daily.

        2. bonkers

          Re: Hmm. 65 000 000 people. 1 000 000 col limit*

          No word of a lie, just a year ago, mates boss sent out a late email saying that he'd had to stop working on the spreadsheet, because the batteries in his calculator had given out.

          Seriously! - he was adding-up the figures and typing them in over the formula.

          1. Eclectic Man Silver badge
            Happy

            Re: Hmm. 65 000 000 people. 1 000 000 col limit*

            @bonkers: Could you introduce your friend's boss to the the Register, please? He sounds like someone with a whole host of excellent 'Who Me?' stories.

            1. bonkers

              Re: Hmm. 65 000 000 people. 1 000 000 col limit*

              "mates boss" - who for purposes of anonymity, we shall refer to as "Matt Hancock"

          2. YetAnotherLocksmith Silver badge

            Re: Hmm. 65 000 000 people. 1 000 000 col limit*

            He probably realised the numbers didn't actually add up, and was checking them.

            That's the thing about spreadsheets - you can hide the maths away across several pages, and get any result you want!

            1. 080

              Re: Hmm. 65 000 000 people. 1 000 000 col limit*

              "and get any result you want!"

              So it is good for statistics then?

              1. AlbertH
                Joke

                Re: Hmm. 65 000 000 people. 1 000 000 col limit*

                "and get any result you want!"

                So it is good for statistics then?

                .....especially "Government Statistics"

          3. MOH

            Re: Hmm. 65 000 000 people. 1 000 000 col limit*

            I worked with a self proclaimed "senior BA" who was testing some dev I'd done on a mainframe application.

            His method was to copy figures off the screen and type them into his calculator, then type the results into an email if they were wrong. I *think* he may have also been writing them down in the interim.

            We hit a production issue shortly after going live because, of the 6 bugs he'd sent me in a week:

            - one was an actual bug which I fixed

            - three of them were typos due to him transposing digits somewhere along the way

            - one was a fairly straightforward case working as intended which he misunderstood

            - the last was an edge case which only happened because of the sequence of events he'd followed, and I couldn't reproduce it. I should have looked harder but assumed it was another typo, but then again so did he when I told him I couldn't find it

    3. macjules

      Re: 'spreadsheet software as "human middleware" in the sector'

      Apparently all data is sent to PHE as a .csv dump file. PHE then open those files into Excel (so far so good - xlsx can handle that ok) but they then have to save the file as an Office 2003 .xls format for uploading into the Track and Trace database.

      So is it the developers of the Track and Trace app and database (Zühlke Engineering?) who are actually to blame. NHSX did raise the issue when they realised that the stumbling block was not being allowed to directly upload the csv files into the database. Therefore not a Matt Hancock problem but definitely a Dido Harding one, or the poor bastard she has handed the job over to as she runs for the door.

      1. EnviableOne

        Re: 'spreadsheet software as "human middleware" in the sector'

        csv is more the middleware used by the NHS, Excel is just a convinient viewer of csv data, as so many of the systems can't talk a common language, you get the system to output a nice csv, that you can get an integration engine to translate for other systems to consume.

        so why the track and trace dashboard can't handle the raw CSV files I dont know

        1. Jimmy2Cows Silver badge

          Re: 'spreadsheet software as "human middleware" in the sector'

          Laziness? Incompetence? Ignorance? Gross negligence? Poor requirements definition? Poor planning? Poor design oversight?

          All of the above...?

        2. keithpeter Silver badge
          Windows

          Re: 'spreadsheet software as "human middleware" in the sector'

          Moodle is an open source/free online learning/course management tool written in php. It can get student data in various ways, including the upload of a csv file. One has done this with many more students (one student per row, natch) than currently the issue. Moodle has been around for a decade or a decade and a half.

          One assumes that a modern hipsta web app would be able to cope with the direct upload of a csv file...

    4. Anonymous Coward
      Anonymous Coward

      Re: 'spreadsheet software as "human middleware" in the sector'

      Uncontrolled spreadsheets production of personal information is the reason that the worst data protection offender is the GOV.

      How can they say anyone has can be forgotten when data is not centralised nor accounted for once it becomes a personal file.

      There has been an increasing reliance upon MS products within GOV and business and that has resulted in a complete lack of control of the data. They should scrap off the shelf software and limit data access to a central secure database that doesn't allow offline viewing or local replication or print out.

      How can the data be under control when the policy is "you havent got a copy? here let me send you one"

      1. GreyWolf

        Re: 'spreadsheet software as "human middleware" in the sector'

        the policy is "you havent got a copy? here let me send you one unencrypted by snail mail"

        Fixed that for you.

  2. error 13
    Facepalm

    The solution is simples:

    https://www.change.org/p/microsoft-increase-the-maximum-number-of-columns-in-excel-to-32-768

    1. Pen-y-gors
      1. Simone
        Facepalm

        Oh surely not...

        W - H - Y - ?.....

        Don't encourage this petition for idiots!!!

        This must the the most stupid idea in the history of stupid ideas ("Dear Microsoft, please double the number of columns in Excel to 32768"). I trust that Microsoft will ignore this, or at least explain why this is a silly request.

        However, when I looked just now, only 7 people had signed it

        1. error 13
          Coat

          Re: Oh surely not...

          Have you had an irony bypass or just had a bad day?

          1. This post has been deleted by its author

          2. Teiwaz

            Re: Oh surely not...

            'Coz government will probably think this a viable solution.

            Don't encourage stupid.

            Ironic too some is a solution reinforced with steel to others.

        2. JDX Gold badge

          Re: Oh surely not...

          16k columns should be enough for anyone.

          1. Rol

            Re: Oh surely not...

            Yep! If you've run out of columns in Excel, you have run out of excuses to avoid learning new tricks.

          2. MOH

            Re: Oh surely not...

            Yep. Why would you record cases in columns, that's just insanse.

            Surely it makes far more sense to add a new worksheet for each one.

    2. DwarfPants
      Devil

      If MS did increase the number of columns, it would not make a lot of difference to the worst use of Excel I get to experience every week.

      Whoever decided putting an error report in Excel where two or three fields are shown on each tab and it takes two or more tabs per record was a good idea needs some percussive reeducation. I truly appreciate the 2000 sheet monstrosity they have created. I hate you even more as you forced me to dust off my VBA skills to turn it into a single readable tab, I thought I had moved on.

  3. Anonymous Coward
    Anonymous Coward

    What I don't get in all this is who decided to ignore the error message? I also don't understand how cases could rise, surely once the file hit the limit you can't add any new data. The cynic in me see this as a distraction from the number of cases and to avoid calls for a second lock down but at least we should all go to the cinema.

    1. Frederic Bloggs
      Facepalm

      CSV?

      If you really want an interchange standard for a transaction log with "lots of rows" that Excel (other spreadsheet programs are available) can read, what is wrong with CSV? With a bit of care it could even count as EDI and bring standards right up to the late '70s.

      Who knows, one could even write a 10 30 line perl python program to add the whole lot to any database of your choice.

      1. Dan 55 Silver badge
        Flame

        Re: CSV?

        Lack of data integrity checks, no way to encrypt contents, the usual CSV madness with fields with dates or spaces in them. To start with.

        As a way of setting up a testing reporting method between Serco and PHE in record time with minimum contract penalties for the government to pay then yeah, it works. Wonderful. That doesn't mean it should be used any more than strictly necessary.

        And whoever signed off on that contract should he taken out back and shot pour encourager les autres.

        1. theOtherJT Silver badge

          Re: CSV?

          The only place where CSV is going to have a problem with content is if it contains a comma, surely. And we have a solution to that it's called \

          1. Dan 55 Silver badge

            Re: CSV?

            Not according to RFC 4180, it says a comma must be escaped by surrounding the field which it belongs in in double quotes.

            1. theOtherJT Silver badge

              Re: CSV?

              Holy shit 2005? I didn't even realize that RFC existed. I've been using it since the 90s and just \ escaping things.

              1. Anonymous Coward
                Anonymous Coward

                Re: CSV?

                The lighthouse csv's were using \ escapes. That broke a lot is stuff (including Excel) because it's not a recognised sequence in the RFC.

            2. Robert Carnegie Silver badge

              Re: CSV?

              I want to know, or rather to rhetorically ask, what happens if the data field contains a quote mark.

              All the software tools I use want to know what an RFC is. Except for our change control system in which an RFC is a change order.

              This one that defines CSV is from 2005 anyway: complying with it now would break all of our carefully managed incompatibilities.

              My preferred way to deal with CSV data is to read the entire line and then apply my own decoder. Generating output, on the other hand, I take pains to remove line breaks and tabs breaks. In my field, nobody really wants these usually, so I just change them to space and to > , which hasn't yet bitten me, but may.

              Another favourite is an external provider which sends us a "CSV" file in which fields are separated with | which is not a comma and also not a problem. Up to now.

              1. Sampler

                Re: CSV?

                Another favourite is an external provider which sends us a "CSV" file in which fields are separated with | which is not a comma and also not a problem. Up to now.

                That's not a CSV, that's a PSV..

                (pipe separated values)

                1. ciaran

                  Re: CSV?

                  In France the default separator is the semicolon, but they still call it CSV.

                  1. Natalie Gritpants Jr

                    Re: CSV?

                    It's French - colon-semi

                    1. WanderingHaggis

                      Re: CSV?

                      Si vous plait, un point virgule, mon vieux.

                      1. 080

                        Re: CSV?

                        Or according to Google a vspdv (variable séparée par des virgules)

                  2. ElPedro100

                    Re: CSV?

                    An alternative interpretation of CSV is Character Separated Values

                    https://www.howtogeek.com/348960/what-is-a-csv-file-and-how-do-i-open-it/

                  3. Charlie Clark Silver badge

                    Re: CSV?

                    That's because the C stands for "character". Can't use commas to separate fields if they're also used for decimals.

                    CSV was hobbled from the the get-go by not using ASCII-31 for separating fields and also by not worrying about encoding.

                2. myhandler

                  Re: CSV?

                  Ceci n'est pas une pipe?

                3. YetAnotherLocksmith Silver badge

                  Re: CSV?

                  Using pipes as separators works really well, at least internally, because you know there aren't any in the data set, and so there's zero risk from commas or apostrophes or quotes! Deals with Mr O'Flynn, and Bobby Tables just fine.

                  (obviously don't trust unvalidated data sources or website javascript validation though!)

                4. SloppyJesse

                  Re: CSV?

                  > That's not a CSV, that's a PSV..

                  >

                  > (pipe separated values)

                  Character separated values? Choose your own character.

              2. Dan 55 Silver badge

                Re: CSV?

                The idea is to escape the separator so you would escape the pipe, not the comma (even though it's not really a CSV so you can do it how you want).

                Depending on the language, Excel can separate fields with semi-colon instead of comma. Yet more CSV parsing fun.

                1. EnviableOne

                  Re: CSV?

                  Actually Excel's default seperator is the tab

                  Tab-delimited CSVs are a PITA, atleast most parsers can take any seperator and text qualifer, even excel manages this....

                  but still why the Deloitte/Zuhlke/Serco system needs office 2007 XLS files and can't read CSV i will never know.....

                2. 6491wm

                  Re: CSV?

                  excel can separate by anything you want which is a PITA if you've used text to columns to split on say ": -" and then forget all about iot and paste something else in and it all splits a bit funny ;o)

        2. Frederic Bloggs

          Re: CSV?

          That'll be the "integrity check" that was both missing and the error message that was ignored in this case then?

          A spreadsheet is something user fiddles with. It is a not a reliable, appropriate, nor scalable means of data exchange. QED.

          Forms of CSV with field definitions, validators and some means of batch (and message) integrity checks is a bog standard - and effective - means of data exchange which, incidentally, underlies far (faaaaaaaar) more of world's data exchange than, clearly, you would be comfortable with.

          1. Dan 55 Silver badge

            Re: CSV?

            Correct. I don't trust them as there's too much ambiguity when it comes to reading and writing them. I know, I have to fix the bugs afterwards.

            And someone is bound to load one into Excel to look at it then save it again for extra fun.

            1. ibmalone

              Re: CSV?

              While I sympathize with the sentiment, it only really applies in the locked room scenario where somebody throws a file over the wall to you. If you're setting up a data transfer you are allowed to talk to the other side in advance, and even if not you can provide a data dictionary to specify fields and how they're stored (date format is the most serious problem, numerical data generally isn't unless there are very large integers, string quoting can be a bit of a pain, but field count checks will catch anything outrageous). In the scenario where you're the person both controlling the generating and reading ends of the process you have the luxury of making it fit your every need. Yes, there are more robust formats, but excel certainly isn't one of them.

              1. Charlie Clark Silver badge

                Re: CSV?

                somebody throws a file over the wall to you

                Happens to me all the time and with variable encoding: utf-8 encoded as latin-1 is not uncommon.

                1. ibmalone

                  Re: CSV?

                  Yes, that kind of mess does exist, but here they should have enough control of the whole process to avoid it. (And it's more of an issue when free text is included, numbers are unaffected by those minor encoding mixups, only if you're off into the wilds of things like ebcdic is it a problem, but I'm not sure you can even run excel on systems that generate that kind of encoding.)

          2. Charlie Clark Silver badge

            Re: CSV?

            CSV is schemaless and doesn't supoort typing or integrity checks.

            1. Sanguma
              Pint

              Re: CSV?

              Have one on me! Nice to know someone else understands the importance of schemas and integrity checks.

              1. Anonymous Coward
                Anonymous Coward

                Re: CSV?

                Well, CSV's can have typing and integrity, you just have to manually code it all yourself which is a lot of work when there are file formats that have pre-written libraries that do all of that for you (and given the quality of the average coder these days, probably too difficult for them to code their own validator). CSV file are still used a lot, for example, the OS elevation data is stored using space separated height sample values (and in other formats as well). This is definitely a PEBCAK failure rather than a file format problem.

                1. ibmalone

                  Re: CSV?

                  CSV handling libraries of course can do that too, why wouldn't they? But this is all a bit of a sideshow, because the argument isn't really that they should have used csv, it's that they should not have used xls.

                2. ArrZarr Silver badge

                  Re: CSV?

                  When I first encountered a .csv, way back when, I thought "I can whip up a validator for this no problem."

                  The next day I looked for a library to do it for me. New line characters in a csv are the real PITA.

              2. Anonymous Coward
                Anonymous Coward

                Re: CSV?

                But the lack of checks isn't CSV's problem. You can send out the world's finest integrity checked data in a perfect schema, and someone can write an import routine that chucks away the checkum, ignores your metadata and assumes metres = feet or pounds = kilos.

            2. Anonymous Coward
              Anonymous Coward

              Re: CSV?

              And so? English is schemaless and doesn't support typing or integrity checks. In fact all digital interchange formats at some point a stream of 1s and 0s without typing or integrity checks. The key is the process that is in place for sending and receiving, sounds like in this case the sender got their process sorted, the receiver cocked up theirs.

              1. Charlie Clark Silver badge
                FAIL

                Re: CSV?

                English is schemaless

                And what do you think grammar is?

                1. EnviableOne
                  FAIL

                  Re: CSV?

                  German has a schema, where there are rules, there are exceptions to those rules, and there are rules for the exceptions.

                  In English, there are rules, they only work less than half the time, and some exceptions are just weird (i before e)

                  1. Charlie Clark Silver badge

                    Re: CSV?

                    Well, yes, but I didn't want to get into the nuances. The fact is that speakers of a language do use grammar as part of the process of speaking the language.

                2. YetAnotherLocksmith Silver badge

                  Re: CSV?

                  Grammar? What do you think it is?

                  1. Charlie Clark Silver badge

                    Re: CSV?

                    Rules, both descriptive and prescriptive, of language: subject predicate object in English; declension of nouns and conjugation of verbs in Latin (so word order is less deterministic), etc.

            3. ibmalone

              Re: CSV?

              CSV in general is schema-less. CSV following a schema you've agreed for data interchange is by definition not schema-less and import libraries will let you specify your expected data types and fields (often trivially). Anyway, the concept only applies here in watered-down form, since we're talking about multiple records of the same form, it's only the field types and order to be specified, not the relation between different types of observation.

              There are definitely other choices which may be better suited to a task, but what csv has over excel is that it's easily generated and read, doesn't have the complexity of multiple sheets and non-data objects, isn't primarily intended to be loaded into software where a field can be altered with an accidental keypress, and doesn't have hard-coded limits on record or field counts (but yes, there are better formats for really big data). And to top it off, xls / xlsx are also schema-less.

        3. J.G.Harston Silver badge

          Re: CSV?

          You encrypt the file by, well, duh, encrypting the file.

          1. Dan 55 Silver badge

            Re: CSV?

            I can see that happening before it's put on a USB stick and taken by train from Serco to PHE HQ.

        4. Anonymous Coward
          Anonymous Coward

          Re: CSV?

          No way to encrypt the contents of a... text file. Please hand in your badge on the way out.

          1. Robert Carnegie Silver badge

            Re: CSV?

            We put the text file in 7-Zip. If you want to go further, you can XOR the data against a one-time pad. Of course it isn't text at that point, but it will be... Or, use ROT13 code. Frphevgl guebhtu vapernfvatyl haxabja bofphevgl. :-)

      2. katrinab Silver badge
        Unhappy

        Re: CSV?

        The problem with CSV is that when you open it in Excel, if there are too many rows, it chops off the ones at the bottom.

        They were receiving the case lists from around the country and compiling them into a single file in Excel.

        1. ibmalone

          Re: CSV?

          But the excel excuse relies on it being used as an interchange format, "oh we were just using it to move data between systems, we aren't so completely incompetent as to maintain all the test records in excel". If it's being combined into a single excel file then that is worse.

          As for csv lacking integrity checks, well, so does excel. Send it with a checksum if you're worried. Yes, excel might produce a warning if a file has been corrupted, but it can't tell you data in a row hasn't been accidentally modified. There is a pretty reliable convention for csv now, and if you control both the sending and receiving end then you can make sure they're in agreement about field data types. If you don't want to do that, R, Stata, SPSS all have their own data formats which are designed for this stuff. Using a data format that is designed to be opened up and monkeyed around with rather than machine processed for data at this scale is a mistake.

        2. Buttons

          Re: CSV?

          In my mind the Covid-19 data was churned by mighty databases processing astoundingly accurate statistics in sparkly data centres. The reality is frightningly mundane and somehow not quite serious enough for the situation.

          1. Sceptic Tank Silver badge
            IT Angle

            Re: CSV?

            Sounds like one place I worked in the 1990's: often if you called the Print Room to find out if they had printed your report (huge stacks of A4 with glue on one side), they would tell you the report is on the binder. I had this vision of a fancy binding machine until I saw it one day. Three steel plates welded together: a floor and two walls all at right angles. The stacks of paper were placed on this device and glue from a bucket was applied with a very huge paintbrush. For urgent jobs there was a Makita industrial hairdryer on hand to help the glue dry.

        3. Ian 31

          Re: CSV?

          That's not a problem with csv, that's a problem with Excel. More accurately, it's a problem with the sort of idiot who thinks Excel is the answer to all their IT issues.

        4. Doctor Syntax Silver badge

          Re: CSV?

          "The problem with CSV is that when you open it in Excel"

          You don't bother opening it with Excel. You open it with an application which has been designed to check for integrity with such things as a declared number of data rows and a proper end of file marker. It fails? Then none of it gets imported and the sender is to to do it again, this time properly, and whatever management processes in place round it are left showing that data from that source is still pending.

          It needs a proper process in place of which the file format are parts S/W and only parts. That needs proper management overseen by someone more capable than a two-time loser of PII.

        5. Loyal Commenter Silver badge

          Re: CSV?

          They were receiving the case lists from around the country and compiling them into a single file in Excel.

          Because, as any fule kno, it is technically impossible to open a CSV file in a text editor, remove the header rows and append it to another file.

          I can only assume that the various CSVs they have to concatenate vary in format or structure in some way, and that someone thought Excel would be the quickest way to sew the things together. As a temporary measure while a programmer spends a day or two doing it properly for you, fine. But WHY THE HELL did they use XLS not XLSX. It's not like it was deprecated ten years ago or anything.

      3. veti Silver badge

        Re: CSV?

        "What's wrong with CSV?" - is that if you double-click on a CSV file, on most Windows boxen, it'll open in Excel. Quietly, and without fuss, deleting everything below row 32768 anyway.

        So that solves precisely nothing.

        1. Jim Whitaker

          Re: CSV?

          And also it will load them into default Excel data types and modify/trim data to fit. Ask geneticists what happens to the gene called SEP1 when loaded into Excel.

      4. thames

        Re: CSV?

        The more important thing is that the data has to come from many different places and CSV may be the only lingua franca that exists for all of them. It's no good looking for the ideal solution if it would take a major overhaul of health IT system to accomplish it. By that time it would be too late and therefore pointless.

        There's a CSV module as part of the standard library in Python. There are also modules which will write MS Excel compatible files if those are desired as the output. It should be possible to write a simple translation filter in a relatively short time. It could include whatever validation and error checking is considered desirable.

        There are still unanswered questions such as why the CSV files had to be converted into the older XLS format rather than the newer one. There's a good chance that there is some other piece of software involved that will import XLS files but not newer formats or anything else useful. That is speculation, but it's a possible answer.

        1. Charlie Clark Silver badge

          Re: CSV?

          I keep coming across CSV files that I can open with csv.reader but can't process properly because they're borked.

      5. ciaran

        Sqlite3 for the win

        I do lots of crazy projects in perl that often involve parsing data coming from mainframes. I've taken to dumping everything into SQLite3 and then working it out from there.

        Its incredibly fast, and SQLite is everywhere.

        Frankly between Excel and SQLite, I know which binary format will be more future-proof.

        1. Charlie Clark Silver badge

          Re: Sqlite3 for the win

          SQLite is indeed useful but for data exchange you might also want to look at HDF, Parquet…

          1. PerlyKing
            Coat

            Re: Sqlite3 for the win

            ...XML <ducks/>

    2. macjules

      From my understanding it was reported right up to the top so that there would be absolutely no misunderstanding that Excel should not be used. The heads of development at NHSX told PHE exactly what would happen if they used a spreadsheet system and sure enough it happened.

      From a civil service point of view every single person involved has done CYA. The sword is now passed to Baroness Dido to fall upon.

      1. katrinab Silver badge
        Unhappy

        It wouldn't be the first time that Typhoid Dido has overseen an IT disaster.

        1. This post has been deleted by its author

      2. Robert Carnegie Silver badge

        I'm feeling, if not confident, hopeful that the organisation I work in will now not require me to use Excel as a database - for about four weeks.

        I am willing to release data by copying it to Excel if that immediately is someone else's problem after that.

        1. Robert Carnegie Silver badge

          So... I wasn't required to do it, but the day after that comment, I got to watch a colleague on video call sorting their table... I can't swear they only sorted one column, but it looked like it.

      3. Tom 38

        The sword is now passed to Baroness Dido to fall upon.

        The Baroness does not fall on swords, she gets promoted.

        1. Roger Greenwood
          Go

          I have already seen it suggested that she is put in charge of organising rubber dinghy supplies in Calais......

          1. gnasher729 Silver badge

            "I have already seen it suggested that she is put in charge of organising rubber dinghy supplies in Calais......"

            That might be a human rights violation, when hundreds of asylum seekers start drowning.

            1. Roger Greenwood

              "hundreds of asylum seekers start drowning"

              I took it that the original commenter was suggesting that it would be likely that dinghy supplies would dry up once she was in charge, thus preventing many perilous crossing attempts.

          2. Anonymous Coward
            Anonymous Coward

            With her organising prevention of asylum seekers? We would be inundated with refugees.

        2. Kane

          "The Baroness does not fall on swords, she gets promoted."

          Doesn't the Peter Principle kick in at some point? Or is that just for us lowly plebs?

          1. Charlie Clark Silver badge

            With the right connections you can't help but keep falling upwards: look at the current PM.

        3. Fruit and Nutcase Silver badge
          Coat

          As a "Circus Act"*, she'll swallow the sword

          * or as Boris would say a "World Class Circus Act"

    3. DS999 Silver badge

      That's easy to understand

      Someone was given a series of steps they are told they must perform exactly as listed without any deviation. If you ask for a robot, you get a robot.

      1. First Light

        Re: That's easy to understand

        A robot that's paid peanuts.

    4. JetSetJim

      IIRC, macro code can be used to suppress warnings and errors. Wouldn't be too hard to imagine some bright spark thinking this solves a few problems on importing a new file.

      This might do the trick, and gets rid of those pesky "are you sure", "file already exists" type messages. Perhaps also the "file got too many rows or columns" one?

      Application.DisplayAlerts = False

    5. Flywheel
      Thumb Down

      The cynic in me says that because (apparently) the missing records were Oop North, they weren't bothered. Hrrrmph.

  4. Anonymous Coward
    Anonymous Coward

    A technical issue was identified

    otherwise known as "a hancock" (or "a dido")

    1. Anonymous Coward
      Anonymous Coward

      Re: A technical issue was identified

      It's a sliding scale from a Hancock, through a Dido up to a full on Grayling. Because there's failing and then there's Grayling.

      1. Frederic Bloggs
        Coat

        Re: A technical issue was identified

        I bet Jeremy Hunt is glad is he safely on the back benches.

        1. shifty_powers

          Re: A technical issue was identified

          You seem to have misspelt his name with a H instead of a C for some strange reason.

        2. OssianScotland

          Re: A technical issue was identified

          Computer User (Non Technical)

    2. Cederic Silver badge

      Re: A technical issue was identified

      More important is that using his name to build the phrase 'cock up' is puerile, unoriginal and entirely aligned to how El Reg has been reporting technology news for what is now decades.

      That self aware lack of shame and willingness to just go for it anyway is admirable. It's a cheap laugh but I did laugh.

      We can only hope that Dido and Hancock give cause for the team at The Register to name them both in the same headline. I look forward to this.

  5. Flocke Kroes Silver badge

    second wake spikes

    Although the article has 'wave' in the title the link to it uses the collective noun for vultures. Make of that what you will.

  6. Laura Kerr
    Mushroom

    A plague on Excel's house!

    Excel has been both a blessing and a curse throughout my working life. A blessing because it's given me plenty of work fixing cock-ups caused by its being (ab)used for data entry and preparation, and a curse because I've had to set about fixing the cock-ups in the first place. Lowlights along the way include Excel outputs breaking batch loads thanks to its 'smart quotes' feature, a major row at a Very Big Bank because the 'test data validation team' were oblivious to the Software from Hell's propensities for arbitrarily reformatting dates, another series of little oopsies caused by the silent conversion of large integers into exponentials and data quality reports going haywire because saved CSV files had loads of empty trailing rows.

    If Serco really are using Excel as a main data store, they deserve to be hung out to dry and permanently barred from getting any future government work, considering the money that Spaffer's spaffed on them. That's absolutely unforgivable.

    See icon for what I'd like to do to Excel.

    1. Charlie Clark Silver badge

      Re: A plague on Excel's house!

      the silent conversion of large integers into exponentials

      You forgot the important bit (sic): converted and truncated: Excel is limited to 15 digits of precision unless you keep it as text. Had this recently relating to customer and card numbers. Oh, how we laughed…

      Still, as interchange format I see fewer problems with Excel because you do have typing and can add data validation, which generally leads to less of a mess than if you give someone a CSV…

    2. Nonymous Crowd Nerd

      Re: A plague on Excel's house!

      One of our clients lost their company's largest contract by selecting all the columns on screen and sorting them by postcode, but overlooking all the columns off screen to the right. Excel offers the user a myriad ways to Hancock-up. The current case, though, must be the biggest ever.

  7. thondwe

    Rows not Columns

    So positive Cases passed 500,000 this weekend, so say two lines per test - date taken, date of result - gives a million - so max rows - from merging all the test centre files - Excel goes bang?

    1. thondwe

      Re: Rows not Columns

      So close, but according to BBC they were using XLS (circa 1987) files rather than XLSX (circa 2007) so limits were 65k rows... we’re doomed!

    2. Pen-y-gors

      Re: Rows not Columns

      From some reports, it wasn't one line (row) per test as any schoolchild would know, but 1 column per test...(max 16K)

  8. Doctor Syntax Silver badge

    The entire team responsible should be sent home to write out "Excel is not a database" by hand, 16,000 times. Each.

    At the very least include a count of results uploaded at the top and check that that number is actually received. For preference also add a specific end of file marker. Ring alarms if one of these tests fails.

    Is it really that hard?

    1. katrinab Silver badge
      Coat

      Excel is not a database^↲

      ^↓

      ^d

      1. ClockworkOwl
        FAIL

        Very Good!

        Now just repeat 15999 more times without error...

        1. katrinab Silver badge
          Unhappy

          ⇑^→^r

          ❌ There isn't enough memory to complete this action. Try using less data or closing other applications [OK]

          1. Anonymous Coward
            Anonymous Coward

            And in that moment, some were enlightened.

            The ones that weren't enlightened at this point were repeatedly struck with the clue hammer and they never repeated the mistake again.

    2. petef

      But they would use a spreadsheet to do that task.

      1. John70

        Should be write it 66,000 times using Excel 97 format.

        Save it

        Give it to their overseer to check

        Tell them they missed 1,000 lines and to do it again from scratch

        Rinse and Repeat until they learn...

  9. cyberdemon Silver badge
    Devil

    VBA implicated yet?

    The only way I can imagine such a howler of an error going unnoticed for so long, is if this poor abused spreadsheet was wrapped in an evil VBA front end, perhaps with the four words of VBA damnation:

    on error resune next
    which translates to "fail silently please"

    but ffs. EXCEL IS NOT A DATABASE! Argh. When will MS Excess & VBA shuffle off into the dustbin of obsolescence..

    1. AVR Bronze badge

      Re: VBA implicated yet?

      They'll go away when there's some kind of useful program used by non-techies which can also be used as a good database. The problem with dedicated database programs is that non-techies don't use them, so they mostly get used by people who are forced to use them. And dumped ASAP when the forcing stops.

      1. Doctor Syntax Silver badge

        Re: VBA implicated yet?

        The problem is trying to do things without getting techies involved. I doubt there's anyone in the techie side of IT - and even a good proportion of management who wouldn't look at what they were using and say "disaster in the making". In this case, of course, the management concerned did not come from that good proportion.

        1. Richard Crossley
          FAIL

          Re: VBA implicated yet?

          They hired Dido Disaster, something like this was going to be the outcome from the start.

        2. Anonymous Coward
          Anonymous Coward

          Re: VBA implicated yet?

          Or is it? Could equally be that this was a job left to the techies, and they didn't look at what their program was churning out and say 'hmmm - this doesn't make sense'? Like the great big hint that if you just diagnosed 500 cases at a University you probably shouldn't be reporting 32 new cases in the same county in your summary.

      2. Rol

        Re: VBA implicated yet?

        Agree. If a program created by the IT department breaks, it remains broken forever, but a spreadsheet you created yourself works until it is broken and then you fix it before the day is out.

        The problem of getting IT resources directed to where they are most urgently needed, is so frustratingly complicated for the user, that you learn to never even go down that route, you just write your own solution, as it is by far the quickest way of fixing things.

        1. Doctor Syntax Silver badge

          Re: VBA implicated yet?

          But they didn't. They didn't even realise it was broken.

      3. sketharaman

        VBA implicated yet?

        Totally agree. I'm not a techie, although I work in tech industry. In circa 1995, I developed a model in Excel to forecast input costs of various components used in a PC. As a sales guy, my objective was to tighten my costing and, hence, pricing so that I could improve my win rates on government tenders. I used ODBC on my Excel worksheet to fetch historical cost data from a mainframe-based inventory management system. I did have to spend an hour or two to learn lookup, pivot table, and other advanced features of Excel but I didn't - anyway couldn't - write a single line of code. It took me a week to develop and test the spreadsheet. When it was ready, it worked as advertised. Our win rate on government tenders went up, which was precisely the purpose of the model. My Excel model became the talk of the company and many others used it for similar purposes in their own LOBs.

        Over the subsequent 25 years, I've had the need for a similar forecasting model a few times, including as recently as last month. Everytime, I've gone to techies as the first port of call to get it developed the "proper way" i.e. with some programming language on some database. Everytime, I've been told it will take "awhile", which is techie-speak for "get lost" to an exec whom they can't rebuff bluntly. If I had to redo such a forecasting model on a tight deadline today, I'd still use Excel. Yeah I know it's 2020 but c'est la vie. Excel is the oldest "no-code" platform I know!

        It's not only me. Per anecdata, 70% of Fortune 500 companies have ERP but 90% of them use Excel to prepare reports for their Board of Directors!

    2. Tim99 Silver badge

      Re: VBA implicated yet?

      "on error resune next" <> "on error resuMe next" - Unless that was on purpose?

  10. M Cts

    I reckon the job was probably left to a biologist in a testing lab as a temporary solution for a couple of weeks, and they did a really good job of it within their IT ability, but the final product never arrived due to moving goalposts etc.

    1. Tom 7

      Never let a badger program your test and trace.They've got form and they've got a valid grudge!

    2. Peter Bennett

      Yeah, I worked in the NHS, I remember a quite senior doctor coded up a shitty timetabling system! My first thought on hearing about it was that a doctor might be behind it :D

    3. Doctor Syntax Silver badge

      If this biologist was in a testing lab with that job the first thing he would do to record results would be to fire up an RDBMS as he has been doing since about 1984.

      However the problem here is to bring batches of results into a single system. In that case the first thing he would do would be to devise a file format with sufficient metadata (i.e. a row count) to check that the received file contained the intended complement of rows so that incomplete files could be instantly noticed. Very likely the format would be CSV. Only after doing that would the S/W be considered.

      Taking time to do a little planning and design up-front pays dividends later.

      1. Eclectic Man Silver badge

        @ Doctor Syntax

        "Taking time to do a little planning and design up-front pays dividends later."

        Are you mad? Nobody does that anymore, it is all Zen, be 'in the moment', react to events, do the 'big picture' but not the details, the purpose of life is suffering, distract the population with disasters to keep them from finding out what is really going on. You need to make an impact to get noticed, otherwise how would you get that much deserved promotion? Planning and design are for wimps.

        Although, now I read that back ...

  11. theOtherJT Silver badge

    If Excel is the answer, you probably didn't understand the question.

    That's my position, and I'm sticking to it.

  12. kpearce

    How appropriate, a spreadsheet was the original 'killer app' after all...

  13. Tom 7

    They're just trolling us now.

    That is all,

  14. You aint sin me, roit
    Coat

    Only 16k records lost?

    Tragic Dido must be losing her touch... she "lost" 10 times that many at Talk Talk.

    Expect further debacles to bring up the numbers to something more respectable...

    1. katrinab Silver badge
      Alert

      Re: Only 16k records lost?

      Only 16k lost records have been found ...

  15. JimC

    Familiar kind of scenario.

    Development time to import data files direct to final system using very flexible and reliable but hugely hassle ridden generic import - two weeks

    Development time using excel as an intermediary due to far more user friendly import routines - twenty minutes

    1. ibmalone

      Re: Familiar kind of scenario.

      Very much disagree, there is database and stats software out there that eats this stuff for breakfast. The problem is putting people on the job who are unaware of this and have been given only a hammer.

    2. Doctor Syntax Silver badge

      Re: Familiar kind of scenario.

      Time taken to sort out the consequences - too long if you happen to have been one of the vulnerable contacts of the unknowing contacts.

    3. CommanderGalaxian
      Boffin

      Re: Familiar kind of scenario.

      Couldn't they just have tweaked their DCOM settings appropriately on their systems and done it all via OPC?

      1. Evil Scot Bronze badge
        Terminator

        Re: Familiar kind of scenario.

        Didn't expect them to use industrial automata.

        Did you mean Object Linking and Embedding.

        1. CommanderGalaxian
          Devil

          Re: Familiar kind of scenario.

          Why go for the plain vanilla stuff? We need OLE for Process Control everywhere.

          Probably could do with a decent SCADA front end too.

  16. Sparkus

    Not to ,mention the potential use of the obscure but oh-so-useful Excel binary format, the .XLSB

  17. Robert Grant

    *Takes a deep breath*

    Stop

    USING

    EXCEL

    1. Tony W

      Re: *Takes a deep breath*

      Nonsense. Many tools can be dangerous, such as a JCB, chainsaw or Neolithic hand-axe. As an engineer I needed spreadsheets constantly. For some types of calculation dedicated software was available but that was even more opaque and inflexible.

      Mind you, it is so easy to get a complex spreadsheet wrong that I was terrified of them. I sometimes checked with dummy data and even constructed the spreadsheet two different ways to check that they gave the same answer, and I did sometimes find errors.

      1. CommanderGalaxian
        Headmaster

        Re: *Takes a deep breath*

        "sometimes" shouldn't come into it. It should always be "always".

  18. Ken Moorhouse Silver badge

    Spreadshit

    Is this the elephant in the room?

    Are we too polite to mention the sh*t word?

    Come on, admit it, we all want to call it a SPREADSHIT.

    1. Martin-73 Silver badge

      Re: Spreadshit

      Indeed, I've never understood what it is actually FOR? Nothing it does can't be done with a calculator and a word processor?

      1. Martin-73 Silver badge

        Re: Spreadshit

        I came here to see if any of the downvoters had attempted to explain what spreadsheets were for. I do admit to having ONE very particular use for one. At great tedium, I set one up to do my end of year taxes. Can't remember how I did it... but i just have to use the calculator to do month by month totals. Any more involved would be tedious. Excel is the single most user hostile application I've come across... That giant '+' instead of a cursor... WHAT?

  19. Ken Moorhouse Silver badge

    So long as Parliament don't ask for it to be sorted...

    But I suppose it will be ok, if they Save As first.

  20. Richard Pennington 1

    Data handling for dummies

    Make up your file, then read back the last record. If the last record is OK, proceed.

    If the last record is not OK, find out what's wrong (and you may yet stay off the front pages).

    And don't ignore error messages. They're there for a reason.

    I'm sure they used to teach this in ICT for 15-year-olds.

    1. Ken Moorhouse Silver badge

      Re: If the last record is OK, proceed.

      Providing there is no duplication.

      IMHO Better to assess "boundary conditions" are not violated prior to import.

      Extra checks need to be performed prior to import? Yes. Oh dear, that's no good then.

      This is the problem with things knocked up in haste, corners are cut.

      1. Doctor Syntax Silver badge

        Re: If the last record is OK, proceed.

        More haste, less speed.

        The time taken to do the job right is usually the shortest in the end.

    2. Socks and Sandals
      Facepalm

      Re: Data handling for dummies

      Trouble is, nearly no-one at senior levels of orgs like NHSX did ICT as a 15-year-old. I looked into their leadership team profile a few months back, and... well, it’s no wonder, is it?

      CTO: BA Economics, MSc Ecommerce Engineering

      Director Digital Transformation: BA English

      Director Technology & Data Strategy: BA Politics, MA International Relations

      National CCIO, Deputy CEO: unspecified degree, first real IT job looks like 2018/2019

      Head of Open Tech: actually seems competent

      CISO & Deputy CEO: BEng Telecoms Engineering, MA Defence Studies

      Director of Delivery and Operations: mBCS since 2016, Said Business School, Major Project Leadership Academy

      Chief Digital Officer: BA Englist Literature, MA Management

      CEO: MA Divinity

      1. Richard Crossley
        Stop

        Re: Data handling for dummies

        Dido Harding:

        - Philosophy, Politics and Economics from Oxford

        - MBA from Harvard

        How did she land her current role?

        - IT skills?

        - Health skills?

        - Political connections?

        1. Jools_UK

          Re: Data handling for dummies

          PPE is the standard education path for MP's

          She surely met them whilst studying

          https://www.theguardian.com/education/2017/feb/23/ppe-oxford-university-degree-that-rules-britain

        2. Eclectic Man Silver badge

          Re: Data handling for dummies

          This sort of thing has been going on for a while:

          https://www.coppertop.co.uk/post/2020/01/13/terry-wogan-can-teach-us-a-lot-about-modern-banking

          "Who is the odd one out: Dennis Stevenson (former chairman of HBOS), Andy Hornby (former chief executive of HBOS), Fred Goodwin (former chief executive of RBS), Tom McKillop (former chairman of RBS) or Terry Wogan (presenter of the Radio 2 breakfast show)?

          If you haven’t guessed already, the correct answer was Terry Wogan – the only one of these individuals with any sort of banking qualification."

          It sounds like the CEO's degree in divinity is potentially the most useful of the lot right now. Hands together, eyes closed ...

        3. Man inna barrel

          Re: Data handling for dummies

          PPE Oxford. It's your golden ticket. Boris was lucky to get into government without it. But I guess Greats will do, if you are a witty kind of fellow.

          MBA is a bonus, so you got the smarts to make cock-ups on a grand scale.

      2. Vometia Munro Silver badge

        Re: Data handling for dummies

        That's a pretty miserable looking list. Especially having encountered some of those particular business school types: they seem to be pretty much a walking attitude problem.

      3. IDoNotThinkSo

        Re: Data handling for dummies

        CEO = MA Divinity

        Pray to God and hope it works?

      4. Kerry Hoskin

        Re: Data handling for dummies

        the ones that worry me are the Chief Digital officer with a BA! and the director of digital transformation another fecking useless BA

      5. Socks and Sandals
        Trollface

        Re: Data handling for dummies

        Oooh, downvotes. Go on, I’m listening.

  21. Nano nano

    you know

    import csv

    with data_file as f:

    reader = csv.reader(f)

    for row in reader:

    1. ibmalone

      Re: you know

      csv code golf, R:

      mydata <- read.csv("mydata.csv")

    2. Oh Matron!

      Re: you know

      Pfffffffffttttttttt

      Pandas FTW!

  22. Jason Hindle Silver badge

    When the only tool at your disposal is Microsoft Excel

    I suppose every problem looks like a nail...

    1. Doctor Syntax Silver badge

      Re: When the only tool at your disposal is Microsoft Excel

      And you end up nailed to the wall.

  23. Len
    Facepalm

    Spreadsheets horror stories

    Another nice story to add to the European Spreadsheet Risk Interest Group (EuSpRIG) horror list.

  24. mittfh
    FAIL

    Error code

    Does this count as PICNIC, PEBKAC, ID-10-T or something else?

    1. John G Imrie

      Re: Error code

      I think its a D1D0 bug.

  25. Anonymous Coward
    Anonymous Coward

    Standard government work....

    Peon creates excel sheet (in their own time) that stores details, then upgrades it a bit so it automatically sends emails, or something.

    Their manager sees it, and mentions it to their boss as a Good Thing.

    Boss tells their boss, everyone gets access to it, and before you know it everyone is using it.

    Original peon leaves for a better job.

    Six months later the thing is overloaded, or stops working, or an Office upgrade renders it useless. Meanwhile it has become "business critical" and there is a Massive Flap (tm) while IT get the blame for it not working.

    Repeat ad nauseum.

    Surprised Excel is being used in this farce I for one am not......

    1. Winkypop Silver badge
      Black Helicopters

      Re: Standard government work....

      AC

      You know too much!!!!!

    2. Anonymous Coward
      Anonymous Coward

      Re: Standard government work....

      i worked in a ship yard back in the mid 90. The whole estimating system was run on Lotus 1-2-3 it was a work of art, but of course only one senior estimator knew anything about it as he'd written it!

  26. CharliePsycho

    Politicians/managers blaming IT? Again?

    Don't think for a moment this is confined to gov.co How/why do you think banks cock it up so frequently?

    There will be a programmer somewhere screaming in an empty room, no BA in sight, no requirements, just a Project manager and a deadline labelled in... Excel...

  27. Jc (the real one)

    This bodes well for the new (beta =fully tested) software being rolled out for customs on the 1st of January!

    Jc

  28. Primus Secundus Tertius

    Old Excel version

    The XLSX file format has been the default for Excel since Office 2007. Were the PHE offices using something as old as Excel 2003?

    Perhaps they were. I saw some very antiquated machines while working on defence contracts. All makes sense to the beancounters at the Treasury, of course.

    I blame the Treasury.

    1. Doctor Syntax Silver badge

      Re: Old Excel version

      I blame the whole lot, from Cummings, via his underling BoJo, downwards.

    2. Richard Crossley
      FAIL

      Re: Old Excel version

      Were the PHE offices using something as old as Excel 2003?

      It works well on Windows XP!

  29. Nano nano

    ignorance breeds ignorance

    If there is nobody at the applicable level in PHE or NHS, that knows the first thing about technical engineering and development, then they won't know to hire people with applicable relevant skills.

    And so it goes.

    1. Anonymous Coward
      Anonymous Coward

      Re: ignorance breeds ignorance

      The problem is there are many within the NHS with the skills and knowledge, even at high levels. They just haven’t been consulted or given the task. AWS, Azure, rdbms, nosql systems, multitude of standard ETL approaches.

      Instead the process was outsourced and PHE starved of funds / bypassed for the most part.

  30. Craig100

    Er, JSON

    Why not use JSON with an API feeding a database or am I missing something? Pretty standard n'est pas?

  31. Nano nano

    Dido (computing):

    see GIGO

    1. Doctor Syntax Silver badge

      Re: Dido (computing):

      A recursive acronym,

      Dido In Disaster Out

      1. Richard Crossley
        Pint

        Re: Dido (computing):

        I wish I'd thought of that. Enjoy -------------------------------------------------->

      2. Ken Moorhouse Silver badge
        Thumb Up

        Re: A recursive acronym: Dido In Disaster Out

        My nomination for Prize Comment of The Thread.

        Doc Syntax shines again.

      3. Nano nano

        Re: Dido (computing):

        You got it !!

        :-D

  32. Danny 2

    Death by PowerPoint

    ...is now mass death by Excel.

    An accountant at one place I worked rigged up a pretty decent Fantasy Football league on Excel, which seemed to me to be what Excel was made for. Next there'll be a Serco contract to convert all the Government notepad documents into wordpad documents.

  33. Lorribot

    Why are these things always reported as an IT or computer glitch or problem when they are quite clearly and human stupidity error by the system architects. developers, Project Managers (who at least have teh excuse that they are just IT users and don't know better but really should) and Developer management who really should not this kind of thing happen.

    Its a bit like saying that car crashes are a highways issue when quite clearly it is always a driver error, yes you can make things more idiot proof but not proof against all idiots.

    1. Doctor Syntax Silver badge

      "they are quite clearly and human stupidity error by the system architects. developers, Project Managers (who at least have teh excuse that they are just IT users and don't know better but really should) and Developer management who really should not this kind of thing happen."

      More likely a human stupidity error then ensured professionals never got near it. Remember this is a government that declared its disdain for experts except on the rare occasion when it realised that it was in over its head.

  34. Winkypop Silver badge
    Facepalm

    Excel by print only

    Worked with a “project manager” who insisted on Excel.

    Only, he refused to accept the data as an .xls file.

    It had to be printed. Only printed.

    1. logicalextreme

      Re: Excel by print only

      One better. Watched a newly-forced-upon-us "Head of IT" (with experience in the NHS, of course) print off half a ream of spreadsheet and ask if anybody had a calculator.

      1. Anonymous Coward
        Anonymous Coward

        Re: Excel by print only

        Sounds sensible . Sense check the spreadsheet and add up a few columns and the totals etc to make sure its doing what it is meant to be doing.Too many IT experts tend to think 'I'm a fantastic programmer and my software has passed all its checks' and don't actually have the ability to eyeball the data and see if it is doing what you think it does.

        1. logicalextreme

          Re: Excel by print only

          No, you misunderstand. It was her spreadsheet. She didn't know that they could do the calculator bit themselves.

          We didn't actually know what she wanted the calculator for when she asked for it. The guy that bit the bullet and responded told her about the calculator application built in to Windows. Then the pile of paper got waved at him.

        2. logicalextreme

          Re: Excel by print only

          But yes, the way you interpreted it is certainly very important. I make a point of getting people to spot check my numbers because not only could I have made a mistake in the implementation, they could have made a mistake in the maths they gave me to implement. I've caught out instances of both by doing this.

      2. Anonymous Coward
        Anonymous Coward

        Re: Excel by print only

        Down vote for the unfounded dig at the NHS. This level of stupidity is prevalent throughout public and private sector. This is demonstrated perfectly by what this is all about. Serco = private. PHE = public and not NHS.

        1. Anonymous Coward
          Anonymous Coward

          Re: Excel by print only

          And actually, to be fair, unfounded digs at Excel and Microsoft. Lots of people saying 'don't use Excel as a database', but the problem seems to be bespoke software that expected ..xls files, and designers who didn't communicate clearly enough that the process of going .csv to .xls, which might (who knows?) have been programmed in all sorts of clever ways at didn't involve Excel, would inevitably cut off the file at 65K rows. In fact it's quite unlikely they did use Excel, because the warning messages re compatibility and truncation when you try and save a 70k row sheet as .xls are pretty clear.

          1. Doctor Syntax Silver badge

            Re: Excel by print only

            "the problem seems to be bespoke software that expected ..xls files"

            You may be assuming something here. It's more likely that the writers of the bespoke S/W were told to expect .xls files and don't complicate things with things manglement don't understand.

          2. logicalextreme

            Re: Excel by print only

            I did think that Excel must almost certainly display a warning when opening a file with more rows than it can grok. Doesn't mean it wasn't ignored though, which would make it the fault of the person who ignored the message or the person that had made them afraid to ask questions.

            The ire at Excel has just exploded in this particular instance because we've all been dealing with that kind of shit for years, inevitably involving spreadsheet cowboying, and we love a good "told you so!" when we get the chance because you can only do that every so often at work without seeming like a prize nob.

        2. logicalextreme

          Re: Excel by print only

          It wasn't a dig at the NHS (I've been trying to get into healthcare for years and have finally achieved that, at a company primarily owned by the NHS), I should have been more specific. It was a dig at legendarily-bollocks Connecting for Health, which is what she was a part of along with the mates and relatives she also brought in and almost sank the company with (and a more general cheap shot at the poor reputation of healthcare IT, which I'm now getting to see the roots of firsthand…but that's precisely why I wanted to get into it!).

  35. Allan George Dyer
    Facepalm

    How to double the problem...

    "For now, we're told, the solution, for want of a better word, is to break the data into two or more spreadsheets."

    Incidently, has anyone mentioned handling of US date formats yet?

    In other news, due to a Y2K problem, all the statistics so far are from the Spanish flu pandemic.

  36. Anonymous Coward
    Anonymous Coward

    I had this rather timely email from the Society of IT Managers in my inbox this morning regarding a webinar, the powers that be had obviously subbed the track and trace app out to a citizen developer!

    Thursday 8 October

    12.30-14.00

    You don't have to be an expert developer. Instead, take skills you already have from Excel and PowerPoint to become a Citizen Developer. Microsoft will:

    • Introduce and compare canvas apps vs model driven apps.

    • Run through UI basics.

    • Let you get hands-on with a workshop exercise.

  37. Ken 16 Silver badge

    It's not the software

    It's the process. Yes CSV have limitations, so does Excel and, God knows, so do people. It has to come down to the process owner not specifying that the process be robust in terms of getting complete and correct data through; ultimately the Minister.

    Fix it, apologise, ensure it doesn't happen again and take the blame at the subsequent enquiry.

    1. Primus Secundus Tertius

      Re: It's not the software

      I used to read Requirements Specifications that duly stated that the software had to be robust. But seldom if ever did I see a Software Design Spec that dealt properly with the requirement to be robust.

      Too many Design Specs were no more than a restatement of the Requirements. Let me explain, with some capital letters, what the difference is.

      The Requirement states WHAT is required. the Design states HOW that is achieved. Except for transcribing maths to Fortran, the two are very different. Most software is about record keeping and organisation, not maths.

    2. Doctor Syntax Silver badge

      Re: It's not the software

      and take duck the blame at the subsequent enquiry

      Fixed to bring into accord with political reality.

  38. Stuart Moore

    I could almost forgive it...

    ... if this had happened in March when there was a rush on and things were having to be set up in a rush. But the fact that 6 months on no one has put in a better process... FFS

  39. Adelio

    Excel

    I remember many moons ago that trying to get Excel at that time to read dates was not easy. (around Y2k time)

    it kept trying to mangle UK dates to US dates, in the end we forced the month as a name "MARCH" THAT worked ok.

    1. Anonymous Coward
      Anonymous Coward

      Re: Excel

      ALL real world date handling is essentially a minefield of horror. Anyone who tells you that it can be sorted easily just hasn't realised how many mistakes they have made. At some point you have to convert idiomatic language to an un-ambiguous code - you can be as clever as you like in your software spec, but that just means you push the problem to someone else who has to convert the data to get it into your system. Still - it helps you say 'not my prob guv'.

    2. Nick Ryan Silver badge

      Re: Excel

      I'm fairly sure that at one point the cretinous piece of shit software parsed non-US dates like this:

      Attempt to parse the date as a USian date. If this fails then try to parse it as a local date.

      In a previous life I had to repeatedly tell a so called smart person to stop storing dates in string variables, and to instead use a date type variable. For months he wouldn't listen and his "amazing" software scripts kept on randomly failing on systems where the date format was not as he expected in his code where he kept on transferring date values by converting them to a string and then back again.

      It's not impossible or always inappropriate to use a string form of a date as a transfer method but it must always be in ISO form and not any localised form at all (including USian for those who repeatedly forget that the rest of the world uses a different date format).

      1. Ken Moorhouse Silver badge

        Re: If this fails then try to parse it as a local date.

        Nice to have it confirmed that this is not something I imagined when encountering this in a spreadsheet I was troubleshooting recently.

        Oh to see the comment-annotated source-code of Excel.

  40. xyz Silver badge

    I once had to..

    Sit in a meeting (circa 2012) between dept A and dept B because dept B wanted to set up a data transfer process from dept A. I had the explicit instructions to find a tech way to deny the request. They wanted excel spreadsheets by email. I laughed, said no and went for a coffee.

    I cannot believe this "muddleware" mentality still exists.

    1. Anonymous Coward
      Anonymous Coward

      Re: I once had to..

      Probably because in the real world Dept B needed the data, and instead of facilitating it the tech guy had 'explicit instructions to find a tech way to deny the request'. And while you were out the room laughing smugly I imagine they just quietly agreed that that was what they would do. They just wouldn't tell the obstructive jobs-worth.

    2. Doctor Syntax Silver badge

      Re: I once had to..

      It would help to know the reasons for denying the request. Security - B not entitled to know? Reliability - data not clean and B want to make decisions based on it?

      1. Anonymous Coward
        Anonymous Coward

        Re: I once had to..

        In which case address those reasons, rather than 'finding a tech reason'. If you want a better way to encourage shadow IT than being deliberately obstructive without explaining why, I cant think what it might be.

  41. CommanderGalaxian
    Paris Hilton

    Why not in Excel?

    At the end of the day, all it's really doing is a telephone-type directory lookup with a couple of bits of information cross-indexed. You'd really need to be a complete idiot to get such a simple thing so badly wrong.

  42. Fading
    Coat

    Excel isn't that bad.....

    Whilst I understand the hate - it works well if you never open csv or txt files with it. Link the underlying data using the "Manage data model" and then anyone will be able to use the functionality in power pivot for the analysis.

    In business Excel is ubiquitous and even the upper echelons know how to navigate around it. So whilst a dashboard linked to a database would be better - in my experience no two managers want the same view of the data (twice) so building individual dashboards can get pretty expensive. At least with the drag and drop functionality in power pivot and with no access to the underlying data you get a bit of peace and quiet......

    1. ibmalone

      Re: Excel isn't that bad.....

      The fundamental problem is spreadsheets mix data, code and presentation, which should be separate, particularly if data is going to be passing through many hands. It certainly has its uses, the ease of setting up some simple calculations or graphs with little training is handy. The problem here is not the use of excel for a report, it's using excel to manage the data storage and transfer.

  43. hamiltoneuk

    We have to be gently with them. THe .xlsx format was only introduced a mere 13 years ago. We have to allow a bit of time for them to catch up. Poor things.

  44. chivo243 Silver badge
    Happy

    Succeeding at failing!

    Dido has it down to a science!

  45. MJI Silver badge

    Seen it wreck data loads of times

    Barcodes are its favourites. After telephones.

    Bye bye leading zeros.

    Long barcodes as strings get turned into exp numbers and do not save as what they were.

    I hate them

    1. Already?

      Re: Seen it wreck data loads of times

      One of our processes involves the client supplying csv data into our process. One particular PM still cannot grasp that opening the data with a double click -> Excel then saving it, is the root cause of data problems, given that all customer UIDs have leading zeroes and this 'quick check, correct a couple of things then save' strips off leading zeroes. Use Notepad. Or Notepad++, or TestPad, or Ultra Edit. Anything but Excel.

      1. Scott 53

        Re: Seen it wreck data loads of times

        "Anything but Excel"

        My answer to "What format do you want?"

    2. Primus Secundus Tertius

      Re: Seen it wreck data loads of times

      IBM had the right idea with Cobol: use binary coded decimal numbers, that can represent a billion dollars to the nearest cent. Excel needs a BCD data type.

  46. Nano nano

    Lockdown loonies

    We had Prof Ferguson's software pulled to bits by the lockdown denialists not long ago; will we now see a "leak" of the Excel sheet update algorithm ?

    1. Primus Secundus Tertius

      Re: Lockdown loonies

      But that is how British public services are run. Bitching, counterbriefing, and personal attacks. Anything except expertise.

    2. Anonymous Coward
      Anonymous Coward

      Re: Lockdown loonies

      Lockdown denialists? ITYF one of them was the guy who wrote it! And even he said it was a system designed to give a rough and ready rule of thumb answer and was based on out of data modelling techniques.

  47. Already?

    For all the [correctly] fine ideas about various better ways of capturing the data into different RDBMS get-ups, the underlying problem is the end user. Data that no-one can see is Write-Only data and next to useless; the idea of issuing everyone who needs to see it - govt bodies, the press, Joe Public - with a means of access to a db is a non-starter, unless an easy to use interface is created - i.e.a web front end or an app. Both cost money and introduce costs, time and risk.

    Excel is the stupidest answer to the problem, but probably fitted the requirements provided at the time. Was end-user access even considered? Maybe not. Maybe it really was a case of beginning with a view to sharing an Excel file around as it worked and was easy, then everybody moved on when cases were still < 1000 / day. Maybe. But in the end, someone with no computer skills would have expected easy access to it.

    1. ibmalone

      By the time it gets to tens of thousands of results the project is big enough that it requires a person in charge who recognises that their desire to see the raw data does not justify using excel as the primary data storage.

    2. Anonymous Coward
      Anonymous Coward

      But this had nothing to do with seeing the data.

      This was about adding several new csv files to the existing data stored in an jnnamed system.

      Then people could look at it.

      Using Excrl to splice it would seem to be a recurring task, done by hand and eye? And likely to use up all the spare attention, leaving none for whether it had worked properly.

      Daily.

  48. Anonymous Coward
    Anonymous Coward

    Use Excel or use Oracle?

    Excel every time.

    The cost of the resultant cockups, data leaks, legal action, repetitional damage and total organisation failure is way, way lower than licensing Oracle to run on a single core anywhere. There are countries whose GDP is less than the money required to run an Exadata 'appliance' over a long weekend; the UK for starters. Hell, even a single 'select from dual' has been known to push people to financial ruin.

    1. Sanguma

      Re: Use Excel or use Oracle?

      Even better - dust off the GPLed release of IngresII and write a workable front end for it. Make it look like a spreadsheet but include a parser to turn a set of common elements into relations/tables and be fastidious about that so that they get the idea and don't misspell everything to D1D0 ...

    2. Anonymous Coward
      Anonymous Coward

      Re: Use Excel or use Oracle?

      hell yes! The licensing cost of Oracle is a total sh1t show. My old employer, a shared service centre who looks after a city council, has its (the city councils) oracle install stuck on an old IBM SAN as to migrate it off to their hyperconverdge infrastructure would cost a shit tonne as the nodes are dual CPU's with a shit tonne of cores in each. Where as the old IBM compute nodes have far less cores

    3. Doctor Syntax Silver badge

      Re: Use Excel or use Oracle?

      False dichotomy.

  49. Nano nano

    Sleight of hand

    Perhaps Dido is set up as the scapegoat, she cannot fall further in terms of tech opprobrium.

    In which case, what else is going on ?

    1. Flywheel

      Re: Sleight of hand

      When she finally lands, she'll bounce off Grayling who previously occupied her type of role.

  50. TimMaher Silver badge
    Coat

    CSV

    Covid Separated Values?

    Just sayin.

    Mine’s the one with the slide rule in the pocket.

  51. Anonymous Coward
    Anonymous Coward

    Cronyism well out in force

    Stating the rampantly obvious. Company that is known to funnel cash to the Tory party, wins contract to deliver multi-million pound system. And then does it on the cheap, using donkeys-old versions of excel to try and handle big data.

    This is absolutely blatant cronyism, it flies in the face of good tendering practise, it is probably in breach of anti-competitive laws; and some poor schmuck on the end of the chain of all of this is probably an underpaid spreadsheet analyst that either A) knows what they are doing and is powerless to change the direction of the idiot company or B) is utterly clueless, and the parent company should not have put them there in the first place.

    In either case, the fault lies squarely with the cronyism that put the job in the hands of these analysts with inadequate tools. Go ahead, downvote me you peasants. :-P

  52. Eclectic Man Silver badge
    Coat

    The actual problem

    When working as an information security consultant, I often encountered clients whose staff each had their own spreadsheets to help them with their jobs. Some were quite small, others quite intricate*. Doing my good old BS7799, and later ISO27001 reviews, I had to inform the clients that they had major risks with these spreadsheets as they:

    0. Had been built by amateurs,

    1. Had not been tested,

    2. Had not been verified,

    3. Are only 'understood' by the person who used them,

    4. Probably had no error checking, or even type checking,

    5. Were only available to the person who built them,

    and

    6. Were vital to the business.

    In addition, many contained personal information of external contacts, and so should be registered with the ICO. (One client admitted to having 300, yup three hundred, separate sets of personal information each run by an individual.)

    I don't know about you, but surely it is not beyond the wit of woman to create some sort of software 'application' that organises information into a standard form, so that it can be presented in an intelligible form, and maybe even passed to other applications for their processing. A sort of Base where the Data could be kept, and used like, as we Mathematical Logicians call it, an 'Oracle', to answer queries. (Silly me, of course not, I guess I am just ahead of my time here.)

    I'll get my coat, coz, well, it is time for lunch :o)

    *I'm referring to the spreadsheets, not the staff here, although some of them were quite small.

    1. Primus Secundus Tertius

      Re: The actual problem

      Spreadsheets are a disaster for ISO9000 Quality Assurance, as in addition to Eclectic's points, they have no audit trail of the design and its changes.

      But that is the difference between software you write for yourself, and software that will be used by others. It is also the difference between sandwich student software and professional engineer's software.

  53. Scott 53
    Facepalm

    Ah, CSV and Excel

    I remember a conversation with a "data analyst" at a major insurance company, who was meant to send me a .csv but sent an .xls instead. Easy mistake to make, sent an email to remind him and got a .csv by return. All good so far, but the file wouldn't load. When I opened it in Notepad++ to check the format, it became obvious that what the "analyst" had done was to rename the original spreadsheet file... Because they were a customer I had to limit my sarcasm in my next message.

    1. Primus Secundus Tertius

      Re: Ah, CSV and Excel

      I had a colleague in a small charity who did not realise that a .ods file was a Libre Office spreadsheet rather than a renamed instance of a csv file.

  54. Anonymous Coward
    Anonymous Coward

    Design

    So who signed off on a 12 million contract and then provided a spreadsheet as the solution?

    1. CommanderGalaxian
      Coffee/keyboard

      Re: Design

      Billion.

      1. Anonymous Coward
        Anonymous Coward

        Re: Design

        That's even worse...

        So where has the money gone?

    2. Brewster's Angle Grinder Silver badge

      Re: Design

      Million? It's 12 billion. That's more than two aircraft carriers (£7,6bn). Or the price of about one F35 to fly on them...

  55. Cynic_999

    Fixed

    The government is now using blockchain.

  56. Security nerd #21

    Whilst bypassing the fact that most accounting organisations & banks rely on Excel to massage / produce their company official financials (so you can really trust what you see - not) - isn't the real elephant in the room the fact that all this data is now stored on x number of laptops, network file shares / Teams / DropBox / Google Drive etc etc

    Nice small data exposure to be considered - but I guess that GDPR / DPA has been forgotten about in this strange times

  57. Spiz

    All the very valid stuff about never using Excel as a database aside, has anyone else picked up on the fact that a SINGLE record is split over MULTIPLE rows?

    A record is a bunch of data in specific fields, right? Why is that being split over multiple rows?

    Surely this is just adding to the list of possible failures in processing this data.

    1. ibmalone

      Yes, that's bizarre. I have a couple of thoughts, which may well be wrong:

      1. long data format that records different value types to different records. Maybe nasal and throat swabs reported separately for example and each patient therefore has a separate row for each with different test type.

      2. Fixed column width format where multiple rows are needed for any field that requires a line wrap. No, you shouldn't need to do that in excel.

  58. Screwed

    On a computer with pretty much no graphics software, particularly no vector graphics, Excel has been surprisingly good for "drawing".

    But if anyone ever chose it for the purpose, they should be fired from a petard.

    Anyone remember batch processing systems? With their End Of File markers? Even record count records?

    1. EnviableOne

      Excel org charts are easy to read, easy to share and everyone can edit them

      if you use the right tool (Visio) most people can't view the natrive format, the file sizes are huge, and no-one can edit them

  59. Snapper

    As the last generation (or two) were taught 'IT' by way of Word, PowerPoint and Excel by teachers who do not know how to supply a template in A4 format rather than US Letter, I would say that the Government Education Ministers who sucked up to Mr. B. Gates to get cheap software in the 1990's - 2000's are culpable here.

    I have twice seen different people fresh out of University compile a list of numbers in Excel, then use a calculator to type in the total at the bottom.

    I have also seen a famous fashion design studio use Excel to create an A4 catalogue of hi-res photographs, then try and send the resulting 85GB file via email.

    Yes, these facts are connected!

  60. keith_w

    Spreadsheets are why we have jobs!

    I trust that everyone remembers that one of the earliest and most popular applications for Personal Computers was Visicalc, closely followed by SuperCalc and then Lotus123 - allowing accountants everywhere to instantly recalculate everything, creating huge macros to do the things they like to do - and creating a huge market for PCs.

    1. Screwed

      Re: Spreadsheets are why we have jobs!

      Certainly do!

      The first time I wrote a seriously useful spreadsheet (in Lotus 123), I found it took up to about twenty minutes to recalculate. Whilst it was worth it for the job it did, the actual data content was tiny. Little enough that you could see every number on the screen at once. No macros at all. Just formulas.

  61. B4PJS
    Facepalm

    The official term - Spreadmarts

    The bane of the Business Intelligence professional everywhere.

    Having been in this industry for a long time now, xls/x has been the bane of many projects I have been on. But yeah guessing xls was the lowest common denominator for getting this thing to run properly on antiquated systems.

    CSV's are great, except for when they are not. Even getting Databricks to correctly read files can be a pain in the rectum. Give me a day or two with their ETL processes and I could prob build a decent platform with pretty power bi reports (I would farm out making the reports pretty to a colleague as all I am good at is tables ) and a robust etl pipeline with all checks and balances in place.

    Ah well, back to my day job of breaking Azure, and then bugging the product teams as to how and when they can do better than me with their own products

  62. codejunky Silver badge

    Ouch

    PHE hasnt done very well with this pandemic. Wasnt the gov gonna bin it a while ago?

    1. Anonymous Coward
      Anonymous Coward

      Re: Ouch

      Yeah blame PHE, who were created by the the Tories in the coalition government as a centralisation of a large network of regionally accountable health authorities and a number of central agencies. A reorganisation that was focussed on 'cost saving' and 'efficiency' - not on public health. A reorganisation that transferred many responsibilities to local authorities, but without concomitant transfer of resources, just before they came under the austerity spotlight. Don't blame a dilettante government who still think that the best way to solve a problem is to make wild irrational promises of pie in the sky or announce a reorganization with no detail, no fore-thought and no timetable (but a cushy job for Dido Grayling).

      1. Sam Haine

        Re: Ouch

        PHE's predecessor, the Health Protection Agency, was fairly useless during the swine flu epidemic as well.

      2. Captain Badmouth
        Pirate

        Re: Ouch

        The lack of adequate funding probably explains the old software in use. Another manufactured "failure" of the NHS and an excuse to hand out contracts in it's ongoing privatisation.

        Privateers everywhere.

    2. Robert Carnegie Silver badge

      Re: Ouch

      Announced back in August to scrap PHE in due course. https://en.wikipedia.org/wiki/Public_Health_England

      You can imagine what Private Eye is saying: or take out a subscription and you'll know.

      1. codejunky Silver badge

        Re: Ouch

        @Robert Carnegie

        "Announced back in August to scrap PHE in due course."

        Cheers. I didnt know the time frame only that the plan was to cut it. I dont read private eye. I used to but stopped at some point.

  63. Strahd Ivarius Silver badge
    FAIL

    One of the common issue...

    in corporate environments or "managed" ones is that MS Office comes with a database application that you are forbidden to use because of "shadow IT": MS Access...

    So you have to rely on Excel for most of your work when manipulating data...

  64. Nonymous Crowd Nerd

    Counting the Cost

    At the end of the affair - when the dust has settled and the excess deaths finally tallied, it will be possible to allocate a cost to this mistake - a cost in terms of lives lost, reopening delayed and cash.

    For that reason, if not for any other, perhaps it's time that a couple of complacent managers actually take the rap for this and lose their jobs.

  65. A_Melbourne

    Last Monday, the WHO announced that 10% of the world's population had been infected. They tried to spin that as bad news.

    However, that means that this virus is no worse than the seasonal influenza.

    Please stop exaggerating this thing. The PCR test is highly-misleading. Claiming that healthy people are "cases" is pure propaganda.

    "WHO (Accidentally) Confirms Covid is No More Dangerous Than Flu

    Head of Health Emergencies Program “best estimates” put IFR at 0.14%"

    https://off-guardian.org/2020/10/08/who-accidentally-confirms-covid-is-no-more-dangerous-than-flu/

    1. Vercor

      Hmm! https://off-guardian.org flagged as: "Proceed with caution: This website severely violates basic journalistic standards."

  66. Anonymous Coward
    Anonymous Coward

    Liars and Statistics

    So what happened, did they accidentally report the *REAL* numbers and not the artificially-inflated fake numbers?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like