back to article Excel Hell: It's not just blame for pandemic pandemonium being spread between the sheets

The howls of disbelieving, horrified laughter caused by the news of the latest pandemic data cock-up yesterday were well deserved. 16,000 cases lost – purportedly in a blunder involving CSV data, row limits, and an out-of-date Excel file format? In a multibillion-pound, "world-beating" contact-tracing system? Unnoticed for a …

  1. Anonymous Coward
    Anonymous Coward

    Microsoft would read: 'It could look at the damage done and say: "We can do better."'

    as

    "We can do better, we have not yet damaged the world enough. Let us redesign and add even more bugs to the product and gaze upon the carnage."

  2. kfpm

    Relax...

    Ok, so you don't like Excel. You've made that perfectly clear. Like most IT tools, languages, there are quirks, nuances, whatever you like to call them. Excel serves a purpose. I hate to use the phrase 'in skilled hands', but that's the important thing. If you know what you're doing, Excel is a very helpful tool. It's not a corporate RDBMS replacement, but, unfortunately, that seems to get overlooked by those who want quick and dirty, and cheap, solutions to their problems. And that's why it's out there, and will stay out there.

    1. Dave314159ggggdffsdds Silver badge

      Re: Relax...

      While I agree with most of what you say, it's worth noting that almost all the actual uses of Excel in practice are as a simple spreadsheet. Which is undeniably useful, and something Excel is pretty good at. That's why it's ubiquitous.

      1. Yet Another Anonymous coward Silver badge

        Re: Relax...

        > almost all the actual uses of Excel in practice are as a simple spreadsheet

        Almost all actual uses of Excel are as an ETL (extract-transform-load) tool.

        ie to download information from one app, reformat/reorganise it and send it to some other app - and it's extraordinarily bad at this - in ways that most users don't know.

        Hence all the genes misclassified as dates, dropped initial zeros in phone numbers, bad handling of embedded commas etc

        1. theModge

          Re: Relax...

          Google made a very fine tool for this, then uncharacteristicly open-sourced it rather than killing it: https://openrefine.org/

          I used it to mark data up semanticly as RDF, but it can get data from damn near anything and turn it into damn near anything else.

          1. Arthur the cat Silver badge
            Happy

            Re: Relax...

            Google made a very fine tool for this, then uncharacteristicly open-sourced it rather than killing it: https://openrefine.org/

            Up vote and my thanks for pointing that out. I hadn't seen it before.

            1. 2+2=5 Silver badge
              Joke

              Re: Relax...

              > Up vote and my thanks for pointing that out. I hadn't seen it before.

              Me neither. BTW, does it have an export to Excel option?

            2. SGJ

              Re: Relax...

              Ditto. I hadn't come across Google/Open refine before - it looks v. interesting

            3. theModge

              Re: Relax...

              Happy to help; I found it for my PhD but I've used it on a lot of projects since

          2. Agamemnon
            Pint

            Re: Relax...

            Oh Wow! Thank You!!!

            1. TRT Silver badge

              Re: Relax...

              So many people say that! It's soooooo much better than excel.

              You know that advert with someone mowing the lawn with an SIV? That's Excel that is.

          3. Johnnie

            Re: Relax...

            Thanks ever so for that.

          4. Wayland Bronze badge

            Re: Relax...

            I used a tool called Monarch. This took printed (text saved to a file) reports from old mainframe type systems and turned them back into tables for import into a database. You had to create a template over the report to tell it which bits where page headings and table headings and actual tabular data. 20 years ago we use it to put purchase orders from a 1980's minicomputer onto a web portal.

          5. Korev Silver badge
            Pint

            Re: Relax...

            Another thanks from me -->

        2. c1ue

          Re: Relax...

          Sorry, but the world isn't made up of data scientist wannabes.

          People actually use Excel for things like modeling, tabulation, accounting and so forth.

          1. Dr Paul Taylor

            If your only tool is a hammer

            People actually use Excel for things like modeling, tabulation, accounting and so forth.

            then every problem looks like a nail.

            If you have tabular data -- where the rows and columns have clearly established meanings as numbers or strings -- then ok use a spreadsheat.

            Otherwise use an appropriate tool.

            1. Norman Nescio

              Re: If your only tool is a hammer

              If you have tabular data -- where the rows and columns have clearly established meanings as numbers or strings -- then ok use a spreadsheat.

              Nope. Excel cannot even cope with tabular data where you want to use logical operations across columns. Give me a count of instances where field 1 has value w, field 2 has value x or y and field 3 has value z, and field 4 is not any of a, b, or c. Loop through all unique values in field 1. It is possible to do with some pretty hairy expressions, but if you are trying to produce a summary tab that works on data in other tabs, or, heaven forfend, external files, you are in for a world of pain.

            2. c1ue

              Re: If your only tool is a hammer

              Wrong.

              Excel is powerful because Ajax allows it to reach out and access/do all manner of things.

              I've created an Excel spreadsheet that would update, from a web page, the prices of my mother's 300 stocks, because her online brokerage account was so crap to understand.

              How else would you do something like that? Particularly 20 years ago...

              Yes, you can hire a programmer to slap together something custom - but what a waste of time and money.

              Sure, it is easier to work with data these days particularly if there is an XML or similar type feed - but businesses existed before this and the Excel pages are an IT artifact just like the XP laptops fronting 15 year old Xray machines.

      2. Marshalltown
        Pint

        Re: Relax...

        I have to disagree. It is more or less OK as a data entry system, if you're not very fussy. But once you start to do anything practical things can go pear shaped in a blink. I was encouraged long ago in the dawn of Windows to use Excel and "shut up." So, simple statistics - came with an add-on set of statistical tools. What could be difficult about calculating the standard deviation of a set of artifact lengths? It is after all simply the square root of the variance, which is mean of the summed squared deviations. There aren't many. I could do it easily by hand, or right there in the spreadsheet. But there's a tool. So - what's this? A negative variance? That can't be right. How can that happen? There's no place for the number "i" in a simple sum of measures. I ran the calculations by hand and there the error was again! Square all deviations, they look right, sum them, still look right, divide to get the mean (the variance) and- where did that negative sign come from? I switched to R right there and then. Later I was told that the problem was probably due to some underflow or overflow condition. All I know is that impossible numbers are no use.

      3. Norman Nescio

        Re: Relax...

        While I agree with most of what you say, it's worth noting that almost all the actual uses of Excel in practice are as a simple spreadsheet. Which is undeniably useful, and something Excel is pretty good at. That's why it's ubiquitous.

        A good, simple spreadsheet would be useful.

        Excel is not a good, simple spreadsheet. It has needless 'helpful' complexity, such as interpreting numbers as dates and applying other formatting rules in difficult to reverse ways. Inserting new, and cutting and pasting rows and columns can have unexpected and unintended effects that are not immediately apparent. Referencing data in external flat files and other spreadsheets is incredibly painful, so you are forced to take unmaintained copies of masters and import the data.

        It is ubiquitous because it comes 'free' in Microsoft Office, not because it is any good, but it is just functional enough to prevent people looking for better choices.

        1. EnviableOne Silver badge
          Windows

          Re: Relax...

          It is ubiquitous because it comes 'free' in Microsoft Office, not because it is any good, but it is just functional enough to prevent people looking for better choices.

          This is the crux of the issue, microsoft does make better choices Access, SQL, Project, Visio, PowerBI, Dynamics, etc. but they cost extra and not everyone has them, and Excel just about does the job, and is nigh on everyone has it.

          Like Word is no longer just a Word Processor, Excel is no longer just a Spreadsheet, it has small parts of all the specialist tools shoehorned in, to make it just usuable enough that finance make you use it instead of paying for the right tool for the job.

    2. Dan 55 Silver badge

      Re: Relax...

      This is a pandemic. It is serious stuff. The non programmer should ask a programmer when they need data crunching done, not do the car-racing equivalent of saying, "yeah, I can do that too" and driving their Ford Fiesta out onto the Formula 1 racetrack and start racing with them.

      The programmer may say "you can use Excel for a while but I need the budget to do a replacement now because one day this will fail and you probably won't even know", that would be better than the non-programmers not knowing, congratulating themselves on a job well done getting the Ford Fiesta out onto the racetrack, and driving the car off the curve and into the audience.

      I used a gratuitous car analogy, sorry.

      1. Yet Another Anonymous coward Silver badge

        Re: Relax...

        >The non programmer should ask a programmer when they need data crunching done

        Yes, cos the intern at the NHS in charge of getting data emailed from 17 incompatible NHS trusts into one report is "empowered" (sorry) to call up Crapita and demand a custom applications

        1. Loyal Commenter Silver badge

          Re: Relax...

          Yes, cos the intern at the NHS in charge of getting data emailed from 17 incompatible NHS trusts into one report is "empowered" (sorry) to call up Crapita and demand a custom applications

          No, they are absolutely not, but the chain of manglement above them is absolutely responsible. Possibly just their immediate manager telling them to "get the job done I don't care how," or maybe right the way to the top of the organisation, depending on how ham-strung the intermediaries are by political edicts.

          Someone is to blame, however, and it isn't the poor intern.

          1. Anonymous Coward
            Anonymous Coward

            Re: Someone is to blame

            Someone is to blame... Dido Harding

            1. The man from Arnold

              Re: Someone is to blame

              Dido, for all her other faults, isn't running PHE.

              1. Loyal Commenter Silver badge

                Re: Someone is to blame

                Indeed. With the utter incompetence of the Current Lot, it's easy to forget that the Previous Lot (who happened to include some of the Current Lot) were also pretty bad. And PHE was put in place by them, no doubt with some stooge at the top who is there not because of their competence, but because of their political connections.

                And we think Russia is corrupt...

              2. Anonymous Coward
                Anonymous Coward

                Re: Someone is to blame

                I thought DH was put in charge of track and trace - which was why she was facing the cameras when it was launched. I also thought that she was inheriting PHE within a broader remit. However, not living in England, I tend to get left behind in the case of change.

              3. EnviableOne Silver badge

                Re: Someone is to blame

                She is, despite Duncan Selbie still being notianlly head of PHE, PHE have been rolled into the new National Institute for Health Protection (NIHP) which She is Interim head of....

                But still the blame has to land at the feet of Andrew Lansley and the Health and Social Care Act, that ripped the Public health boards apart and gave the responsibilities to local councils, without the funding, and deposited the central parts in PHE, so it was basically a controll centre for something that no-longer existed.

                Maintaining a Public Health infrastructure is expensive, but when an epi or pan demic comes along it repays that in spades, but if you dont have one, it starts to look like a good line item to cut in a budget

            2. Kane Silver badge
              Joke

              Re: Someone is to blame

              "Someone is to blame... Dido Harding"

              Twat.

              Sorry, not you; I have a slightly nervous reaction to the words Dido Harding, in that I involuntarily type "Twat" whenever I see it on screen.

              Twat.

              See? Did it again! I really should write that custom dictionary that converts her name onscreen into Dildo Hardening or something to prevent this from happening in the future.

              Twat.

              Dammit, that didn't work either...

        2. Doctor Syntax Silver badge

          Re: Relax...

          In real life this will have gone through all sorts of very important people an NHSX who then said "use Excel because that's what we know.".

          1. Evil_Goblin

            Re: Relax...

            Allegedly NHSX forbade the use of Excel "on their side" but once it got to PHE it was Excel all the way...

            1. EnviableOne Silver badge

              Re: Relax...

              Its Not PHE, its the Delloite/Serco Test and Trace Dashboard that needs the excel 2007 format stuff

        3. Dan 55 Silver badge

          Re: Relax...

          If they don't have someone they can call for basic IT advice when designing a project, they're doing it wrong. Really this should be the BA's domain.

          (Didn't downvote.)

      2. Persona Silver badge

        Re: Relax...

        The programmer may say you can use Excel for a while but I need the budget to do a replacement now because one day this will fail and you probably won't even know

        The programmer then went on to code the Post Office accounting system and some users were convicted of fraud when the money that supposedly had been collected was not there.

        There are bugs everywhere. Government IT solutions have a habit of taking years, getting cancelled before they deliver anything and when they do deliver the results are still wrong.

        1. ibmalone Silver badge

          Re: Relax...

          It's a good point, but it shouldn't have to be like that. Horizon in particular started out as something else that was Too Big to Fail (before that was cool) and got shoe-horned into its final role.

        2. A.P. Veening Silver badge

          Re: Relax...

          Government IT solutions have a habit of taking years, getting cancelled before they deliver anything and when they do deliver the results are still wrong.

          For government IT solutions the available options are working, on time and within budget, just pick any zero out of three.

      3. T. F. M. Reader

        Re: Relax...

        The non programmer should ask a programmer when they need data crunching done

        I imagine a reality satisfying the following set of requirements and specifications.

        Matt Hancock needs to make presentations of "stuff" to various audiences - PM, Cabinet, MPs, press, public at large, you name it. This is the primary requirement, any data analysis needed to understand and deal with the virus spread is by definition secondary.

        To satisfy the primary requirement a flunky, probably with a title like "political advisor", decides what slices of data should be presented. This varies between presentations, probably many times a day. There may be more than one flunky - a committee - doesn't matter.

        Another flunky needs to create graphs and bar charts and pie charts on demand, for each presentation, under pressure, because the first flunky tends to change the decisions with very little time to spare. If there is a committee there will be even less time. There is absolutely no way a programmer can be engaged for this. The only tool the flunky knows how to use is Excel. This means that whatever methods of collecting, storing, and processing data are used upstream the data must be available in Excel so that the geeky flunky will be able to produce new charts many times a day.

        Upstream, there is a clear understanding what the primary requirement is, so Excel is chosen as a major, likely the primary, possibly the only way to collect and store data. Someone figures out that most secondary data processing tasks are not terribly complicated and can be done in Excel. No one knows that Excel's numerics and statistics are "quirky", and if anyone does they are told it is not an important enough consideration. "Best practices", as in, "everybody uses Excel", are probably mentioned.

        Cynical? Moi?

        1. Flywheel Silver badge

          Re: Relax...

          Matt Hancock needs to make presentations of "stuff" to various audiences - PM, Cabinet, MPs, press, public at large, you name it

          We have the British Computer Society (BCS) with a huge diversity of experience and knowledge among its members both in the UK and worldwide. We need Hancock to appear before that committee and see if his ideas hold up.

      4. Flywheel Silver badge
        Facepalm

        Re: Relax...

        It's the B-word: Budget. If Baroness Harding can come out with nonsense like "We weren't expecting a [global] pandemic to create so many cases of illness and death" then she and Hancock certainly won't have allocated a budget for software and/or development. I can imagine Hancock telling his data minions to "upgrade to a later version of Excel with more rows" when the original version ran out of capacity.

    3. Anonymous Coward
      Anonymous Coward

      Re: Relax...

      The sad thing is there are quick and dirty solutions out there...a simple bash script could have done what this fuckwit tried to build in Excel.

      The fact of the matter is, the chap that implemented this solution was technically incompetent on every level.

      I suspect the vast majority of us here could parse and process a CSV in any number of ways and it would take us a while to find someone that leapt straight at Excel.

      Excel may have it's purpose...but as a batch processor for CSVs it is so ill suited that literally every techie everywhere hearing that it was used for this felt dirty.

      I have no idea what they were parsing into, but I'm reasonably confident whatever it was has a command line tool available that you can pipe to.

      cat covid_crap.csv | while read line

      do

      read -d, column_1 column_2 < <(echo $line)

      # DO SOMETHING WITH THE DATA HERE, PIPE IT TO A SCRIPT IF YOU LIKE, IDGAF

      done

      Admittedly this isn't the most efficient way to parse a CSV, but it'll snarl through hundreds of thousands of rows in no time flat.

      There's absolutely no need to have a fully featured GUI application in the middle. Zero.

      Even if you had to write a Python or PHP script with a database connection to handle the data once it's parsed, it would take less time than building an Excel VBA macro.

      What the Excel solution reeks of to me is the handy work of some academic intern using an example snippet from a text book somewhere. Nobody with actual real world hands on experience handling files larger than 1mb would ever look to Excel as a solution. Only someone unaware of it's limitations and a lack of knowledge outside of a Microsoft ecosystem. 65k rows in a CSV isn't even that big, I've kicked around CSVs with hundreds of millions of rows before with relatively little trouble...it never once crossed my mind to convert those CSVs into something else before I parsed them...I just parsed the CSVs.

      The only situation I can see that Excel might be necessary is if there is some wanker .NET developer somewhere in the chain on the backend system hard coded the import to only use Excel spreadsheets for "compatibility" because he has never used CSV files before and had Excel installed on his laptop which enabled him to test with small datasets.

      Apologies to all the .NET devs out there, there probably is a good one somewhere...but come on guys, you're the only developers that see throwing more RAM at a SQL server as being a solution. It's never your crap nested UNIONS and JOINS that are the problem. It's also never the complicated SPROCS performing logic that the client side application should be doing.

      Also, if I had a quid for every time a .NET developer said "well it's NGINX that is displaying the error, the problem must be there, look it says BAD GATEWAY", I'd be a very rich man.

      1. veti Silver badge

        Re: Relax...

        My guess is that, way back in approximately late February, when the new cases per day were in single figures, someone put them into a spreadsheet and circulated it to some bigwigs. They quickly got into the habit of doing this daily.

        Then things blew up.

      2. Christopher Reeve's Horse

        Re: Relax...

        Technically incompetent at every level?

        Yes, your argument is very good, but typically the user of excel has an entirely different area of technical competence. You clearly have a technical competence for Bash scripts, and that's great, but the problem is the overlap of responsibilities and the management of resources, and probably not the technical competence of the original user.

      3. David Nash

        Re: Relax...

        "someone unaware of it's limitations and a lack of knowledge outside of a Microsoft ecosystem."

        I think you've hit the nail on the head there.

        Someone who had to come up with a solution didn't have the knowledge or experience to know that there were better solutions available. "Excel can hold thousands of rows of data and it's on everyone's desktop, what's not to like?"

        Non-techies probably wouldn't even consider that there may be a limit on the number of rows.

      4. tfewster Silver badge
        Facepalm

        Re: Relax...

        @AC, your script fails if there's a comma in a data field, even if it's within quotes (the CSV standard).

        Better to use something that understands the CSV standard, like, um...Excel? In my experience, it tells you if the CSV data hasn't completely loaded ( > 1M rows, so that might only be in later versions).

        I believe PowerShell has an Import-CSV function. I expect there are Linux utilities that can do that too. I'd probably find them useful, as one of my duties involves producing a huge, meaningless report listing which I then parse in a bash shell to pull out useful data.

        Excel is remarkably good at making up for systems that can't produce decent reports, though it has pitfalls. The real WTF is why a 65,000 row "report" is needed - a human won't read it. If it's using CSV to transfer data from a collection system to a proper analysis tool, that's fine. If Excel is your (temporary) analysis tool, just be aware of the limitations.

        Downvote away if you have a ready-to-roll analysis tool for, say, < £500/seat ;-)

        1. Paddy
          Pint

          Re: Relax...

          Jupyter/Python/Pandas will get you further.

          1. tfewster Silver badge
            Facepalm

            Re: Relax...

            Thanks Paddy - It's embarrassing how much time I've spent writing my own bash utilities to import/manipulate/write CSV files instead of just looking for something ready-made!

        2. Anonymous Coward
          Anonymous Coward

          Re: Relax...

          Yes I'm aware. I spent all of 10 seconds on it. The point is, I could use the remaining time the Excel solution wasted to refine and test.

      5. Norman Nescio

        Re: Relax...

        The sad thing is there are quick and dirty solutions out there...a simple bash script could have done what this fuckwit tried to build in Excel.

        Speaking from experience here as a low paid administrator (I have other experience), the locked down laptop does not have bash available, and it is not available in the corporate-approved portfolio of applications available via IT. Neither is, for example, SQLite.

        On many occasions I have been reduced to bringing in my personal laptop to get stuff done that simply wasn't possible using the corporate IT set-up available to end users. Can't do that with health data as GDPR and basic data privacy rules mean I can't move the data out of the 'secure' corporate environment - which is correct, and I would not break those rules. It makes life challenging.

        1. uccsoundman

          Re: Relax...

          You and I (and many others here) are the exception, but being a hiring manager for software development I am constantly astonished at how many CS graduates have never seen a command line interface. And in my company, until just a year ago, even MENTIONING Linux or Unix could get you fired; we're a windows house. So often it's just better to do what you are told and get the wrong answer than to risk your job by doing it right.

          1. Anonymous Coward
            Anonymous Coward

            Re: Relax...

            Fired for mentioning Linux? What fucked up madness is this?

            Is using the word Linux like mentioning Voldemort in a Windows shop?

        2. EnviableOne Silver badge

          Re: Relax...

          Bash is probably better, but you can do just about enough with MS Batch Scripts or Powershell, and even the local user can run those

        3. Anonymous Coward
          Anonymous Coward

          Re: Relax...

          GDPR and privacy controls only apply if the data hasn't been anonymised effectively.

          Assuming the track and trace folks only get personal information when they narrow down to a venue etc, there should be nothing in the data that is personally identifiable.

          BYOD is no better than installing unsanctioned software on the corporate network. Either way, you're circumventing policy.

      6. EnviableOne Silver badge

        Re: Relax...

        Even a .NET dev would use native commands in Powershell to parse the raw csv.

    4. FlamingDeath Silver badge

      Re: Relax...

      Why are you defending micosoft?

      That shitshow of a road accident dressed up as a software house deserves all the scorn it can get

      1. FlamingDeath Silver badge

        Re: Relax...

        If you think their excel is a spawn from hell directly out of the anus of bill gates Arse™️ (Version 3.2, release 8)

        Then consider the amount of electricity wasted on this fucking awful operating system while it attempts to complete an “update”. How many power stations are needed worldwide to accommodate this travesty ?

        Oh, and Dr. Bill want to give as all vaccines, you couldn’t make this shit up

      2. Cederic Silver badge

        Re: Relax...

        Because the fault isn't with Microsoft.

        The fault isn't with the Government, or the people managing that Department.

        The fault isn't with the users that are using Excel.

        The fault is one of IT strategy and governance. You know that users will use Excel, you know that they're going to use its competitors if you try and stop them, you know that they're going to do so because it enables them to do their job quicker and often better, you know that they have little choice because any time they ask IT to implement a solution it costs too much, takes too long and doesn't do the right damn thing anyway.

        So where are the policies and procedures to control its use, where are the data standards and validation to prevent issues like this, where's the training to assure people don't misuse it, why aren't other tools provided that let people do data merges more reliably?

        I hate the prevalence and misuse of Excel at organisations but there's only one part of the company able to do something about it.

  3. Jan 0 Silver badge

    Hurrah!

    You have to grudgingly admire the perpetrators one of the greatest scams. You have to despair of the army of managers that have allowed this overgrown toy to take over the world.

    1. TRT Silver badge

      Re: Hurrah!

      I've had this discussion. The so called IT department that deinstalls things like OpenRefine and insists we can use Excel. *shakes head*

    2. AndyD 8-)&#8377;

      Thirty years ago...

      ... let me say that again - thirty years ago I wrote a report for a major bank, outlining the risks and drawbacks of using spreadsheets for almost anything!

      This was a bank that had very rigorous very very thorough, very s l o w, but almost completely foolproof in-house software development procedures. My reportt was received 'politely'.

      A few short months later they made swingeing redundancies across the IT department and brought in newer, faster fashionable techniques.

      Hey Ho

      1. uccsoundman

        Re: Thirty years ago...

        And I'll be that these newer faster fashionable techniques resulted in massive failures in production, downtime, and lost revenue, if not worse. But it was Agile, so that's OK. Who cares if the customer loses money?

  4. Dave314159ggggdffsdds Silver badge

    This is just a teenage rant from 2000. I'm surprised the author doesn't refer to micro$oft.

    Obviously most of it is hyperbolic verbiage, but the substantive parts are simply wrong. Excel is like mole grips: there are very few situations it's the really correct tool to use, but in an awful lot of others it's good enough to get the job done when you don't have the perfect tool.

    Blaming the tool for someone's idiocy is silly; there are endless examples of idiots managing to idiot even with the right tools.

    1. vtcodger Silver badge

      Downvoted because in my experience Excel adds a variety of bizarre idiosyncracies on top of the numerous normal hazards of spreadsheet use. I'm talking about things that didn't/don't happen in better behaved spreadsheets. My favorite -- the teacher who tried to copy and paste a column of zip codes. Excel inserted the first then copied down as expected. But it quietly incremented each code after the first 90210, 90211, 90212 ...

      1. Aleph0

        Excel only does that incrementing while doing autofill, not copy/paste. Granted, one can inadvertently invoke autofill with a clumsy mouse drag, but the Undo command is there for a reason, isn't it?

        1. Gene Cash Silver badge

          > one can inadvertently invoke autofill with a clumsy mouse drag

          As someone with the motor skills of a dead cow, I say "fuck that shit" - I'm so tired of unexplained strange actions from GUIs.

          That is still broken behaviour. I would not expect any such response from Excel.

          1. Cederic Silver badge

            If only you could hit ctrl-z and undo the clumsy edit.

            Software design is a delicate balance between ease of use for inexperienced users, accessibility and enabling power users. Excel's autofill capability is not broken behaviour, even if you disagree with where they've tried to find that balance.

      2. Agamemnon

        Upvote, you used West Hollywood zipcodes in a snark.

    2. big_D Silver badge

      Having faced many abortions of "systems", "developed" in Excel, I can only agree with Rupert.

      Using Excel as a proper spreadsheet to analyse some figures, fine.

      But I've experienced:

      * Timesheet system for a whole company automated with Excel 4 macros (not VBA, Excel Macros).

      * Forecasting system so complex in 1-2-3 that Lotus threw up their hands in disgust and told my project manager to use a real language!

      * Production completely bypassing an ERP system, doing planning and production in Excel and booking the end result back into ERP.

      * Complex "databases" automated in Excel

      And many more sins.

      1. Daniel von Asmuth

        My Excel skills are limited to typing a simple table, then copying it into a Word document.

        1. Marshalltown

          Simple tables

          In Word and some other current wp's you can type the values in as comma or tab separated columns, select the block and tell the word processor to convert it to a table. You can then the format the table in the wp. Or, you can simply copy a csv or tsv file, paste it into the wp and again convert to table. Vastly better than creating tables using the GUI and less aggravating than dealing with some of the weird incompatibilities encountered occasionally even between MS products.

          1. ovation1357

            Re: Simple tables

            Wow! If this is true then I'm interested. I'm going to be trying this in Word tomorrow

            1. big_D Silver badge

              Re: Simple tables

              It is. It is quick, easy and flexible.

          2. big_D Silver badge

            Re: Simple tables

            That or tab separated. That is the way I've been doing tables in Word for a couple of decades.

      2. Anonymous Coward
        Anonymous Coward

        Yep. Where I am we use a "spreadsheet" to record hardware in racks at the data centre (not a list but cells made to look like the racks!), then when and SSL cert was forgotten and expired someone started up a new spreadsheet to track SSL certs (which is hardly ever checked or updated).

        The other one is a particular gripe of mine - our incident and change management system has a thousand different specific data fields, yet most are irrelevant or unused in favour of copy+pasting some plain text templates into about 5 separate text areas and painstakingly filling in the blanks! For all change requests it's necessary to fill in a 'risk scorecard' which is... You guessed it! A f**king Excel spreadsheet with macros where you answer some multiple choice questions about the scale and impact of the change, press 'calculate' and then copy+paste the risk scorecard' back into a text box in the change request.

        We're paying vast sums for a big name CoTS product and then using it like it's notepad.exe plus Excel to do what should be a built in form.

        It's madness but everywhere seems to do similar horrible things :-(

        1. TRT Silver badge

          I sat in on a session today. Updating 785 network switches across 13 sites with 53 buildings.

          The project was being managed using Excel. The consultant must have spent 60+ hours constructing an intricate balance of pivot charts and look up fields. At £400 am hour no doubt. I watched him live on a call with the stakeholders add in all the unidentified risks they'd missed. The whole thing collapsed as soon as he added a single line. And we pay for it.

    3. Primus Secundus Tertius

      Ye Olde English Proverb

      A bad workman blames his tools.

      [No, not the singular!]

      1. Doctor Syntax Silver badge

        Re: Ye Olde English Proverb

        The bad workman might have chosen the tools.

        1. Eclectic Man Silver badge

          Re: Ye Olde English Proverb

          In the case of the NHS Covid-19 reporting app, I suspect the 'bad workman' was told which tool to use.

        2. Dippywood

          Re: Ye Olde English Proverb

          The 'tools' are normally management...

      2. Loyal Commenter Silver badge

        Re: Ye Olde English Proverb

        A bad workman tries to insert a screw with a lump hammer.

        In this analogy, Excel is the hammer, and the screw is your data.

        1. DJ
          Pint

          Re: Ye Olde English Proverb

          Hammer. Hammer?

          Oh, he means an Irish screwdriver.

          I'll just be going now...

      3. The commentard formerly known as Mister_C Silver badge

        Re: Ye Olde English Proverb

        A good workman uses the right tool in the right way. And knows the difference between good quality tools and cheap tat.

        1. TRT Silver badge

          Re: Ye Olde English Proverb

          Excel is to tools as a Swiss Army Knife is to. Well. Tools. Handy, everyone has one, knows how to open it. And then proceeds to use it to cut down a tree, make planed timber, turn it into a flat pack and build a cabinet. But that must be the right tool as it even had a thing for getting stones out of hooves.

          1. the Jim bloke Silver badge
            Pint

            Re: Ye Olde English Proverb

            I have used a swiss army knife to cut down a tree.

            There was alcohol involved.

            This also led to setting the stump on fire, and driving work vehicles across the smouldering stump. I cant remember just why that particular tree merited such treatment, but it must have been something heinous.

            Generally I only used swiss army knives - and leatherman tools now, to remove branchs or shorten small lengths of timber, but that tree had it coming.

            1. TRT Silver badge

              Re: Ye Olde English Proverb

              Excel is like alcohol too. Perfect for a tiny range of uses like flavour extraction etc. Dangerous when abused. Many people seem to like it. It can become dangerously addictive.

          2. MattWPBS

            Re: Ye Olde English Proverb

            Swiss Army Knife is the analogy I always use.

            You can trim branches with the little saw blade, you can slice up food with the knife blades. If you end up trying to run a lumber business with Swiss Army Knife though, you want to take a step back and think about what you're doing.

      4. Anonymous Coward
        Anonymous Coward

        Re: Ye Olde English Proverb

        Re-arranging that logic, a bad workman is a tool, which means his managers can't blame him. Therefore those responsible for this mess are the managers. Unless his managers are also tools, in which case those above can't blame the managers...and so on.

        It would appear that logically, an organisation full of tools is blameless...we have to look at how they got there...

        Next GE everyone needs a long, hard think.

        1. veti Silver badge

          Re: Ye Olde English Proverb

          They're public servants, which means they're ultimately answerable to politicians. Who are elected by...

    4. xeroks

      upvoted

      upvoted because for decades my goto DIY tools were a medium sized swiss army knife and a pair of mole grips.

      I have other tools now, but they dynamic duo get regular outings.

      As for excel. Microsoft did try to upgrade it, calling the results Access. We all know how that went.

      1. aidanstevens
        Thumb Up

        Re: upvoted

        Upvote for you purely for "dynamic duo" to describe a Swiss army knife and a pair of mole grips.

    5. Dave K Silver badge

      The problem is that MS have added things to Excel to make it more versatile and have enabled these abuses of Excel. It does have its place and in certain situations, it is the correct tool.

      I consider Excel to be a bit like a pair of mole grips with a multimeter, wrench, soldering iron etc. gaffer-taped to the handles. The functionality is sort of there, but that doesn't mean you should use those mole grips for soldering a circuit board together...

    6. Loyal Commenter Silver badge

      ... there are very few situations it's the really correct tool to use, but in an awful lot of others it's good enough to get the job done when you don't have the perfect tool.

      The problem is that whilst being "good enough", it fails in often untraceable and unpredictable ways. So what you think might be "good enough"suddenly stops working or produces errors you don't even know about.

      Like losing the results of thousands of positive coronavirus tests.

      Unless you are an "expert user" of Excel, you may well not even know about the limitations it has that would produce these problems, so you, quite reasonably, might think it is a suitable tool for the job when it is not. This is the real evil of Excel.

      That and lack of testability, reliability, traceability, and security in a half-arsed incomplete programming model. It allows people to think they are being clever getting "calculations" done without even realising that they are effectively doing a programming job, but without the benefit of knowing how to program, and the caveats that would bring, such as the need to define behaviour, boundary conditions, inputs, outputs, etc. and verify and test the behaviour to ensure it is correct. Actually, that - that is why Excel is Evil.

      Oh, and people's thinking that it can be used as a simple ETL tool, with the assumption that if you export something to Excel and re-import it again into $software, opening the file before re-importing it won't suddenly decide to convert half your values into random numbers, or dates, or remove leading zeroes, drop precision from numbers, choke on dates before 1900 (whilst thinking that there was a 29th February in that year) and all other sorts of shenanigans. THAT is why Excel is EVIL.

      I mean, even for its original purpose, of simple accounting spreadsheets, it has the complete lack of structure and control that you would get in any actual accounting software. It might be okay for reporting, with various caveats, but invariably, whenever I have come across it in the 25+ years I have been working with computers, there has been a better or more appropriate tool for the job. In summary, Excel is EVIL

      1. ibmalone Silver badge

        You left out source control (unless it's hiding under traceability). How is this version of the spreadsheet different from that version of the spreadsheet, and why was that change made?

        1. Dave314159ggggdffsdds Silver badge

          Yet another thing that is built into Excel...

    7. Krassi

      Excel is used all the time because the specialised tools don't work ! It is harsh to blame excel for being crap when it is being used as sticking plaster to patch up much more expensive and supposedly advanced database systems that can't do simple (*) tasks like passing data from one place to another.

      (*) far from simple of course, but the objective is simple and easy to ask for.

      1. ibmalone Silver badge

        Apparently excel can't do that either, it just pretends to.

  5. WolfFan Silver badge

    I get it

    It’s Microsoft’s fault. Hmm. I remember when a small, little-known, company named, ah, ‘Lotus’ brought out a far superior product named Jazz. I remember it crashing and burning, for several reasons: it was Mac-only in 1985, it cost $600 (in 1985! Jesus bloody Christ, what were they smoking at Lotus?), it was aggressively and yet uselessly copy protected (at the time the standard Mac floppy was 400 kB, but the Jazz disk 1 was 405 kB, making it impossible to copy... unless you had a copy of CopyIIMac or similar, which copied it in minutes) and, most crucially, it sold under 20,000 copies (see ‘Mac-only’, ‘$600’, and ‘outrageously copy-protected’) while the then brand-new Excel v1 sold over 200,000 copies because while it was Mac-only (until Windows 1 showed up, that is) it cost less than half what Jazz did and was not nearly as user-hostile. Excel also did much less than Jazz, but went on to become _the_ killer app on Macs, Win 1-3, Win 95, Win NT... Lotus canceled a follow-up, ‘Modern Jazz’, which would have been much better and possibly run on Win 1-3. Lotus also didn’t bring out a Mac version of 1-2-3 until 1992, and brought out a Windows version of 1-2-3 far too late. Borland Quatro crashed, burned, and went to Canada, becoming part of Word Perfect Office from Corel... and notably stayed away from Macs after Corel killed the Mac version of Word Perfect. Excel was left as last man standing, with no real competition, and not due to anything Microsoft did. Lotus did try again with Symphony, but by that time it was too little, too late.

    People used Excel because there was no real competition and because it was what they had. If Lotus, or Borland, or _someone_ had put something worthwhile out as competition to Excel, then perhaps things would have been different.

    1. Anonymous Coward
      Anonymous Coward

      Re: I get it

      Only one problem with you story. Thats not what actually happen. I know. Because I was there.

      From the mid 80's to the mid 90's the Mac market was an afterthought in the business software market. It was 95% MS/DOS only. Then starting to make the transition to Windows with Win3.1 but it was Win95 that was the real beginning of the end of the MS/DOS ISV market. I was reading those monthly market share / vendor share channel stats from 1985 onwards. It was part of my job as someone who was running dev teams in exactly those segments. On those platforms. Mac /MS/DOS / Win 3.x

      MS had weak market positions, basically an afterthought, in wordprocessors and spreadsheets until the platform change from MS/DOS to Win 3.x. Multiplan trailed Lotus 1.2.3/ Quattro etc. Word trailed Wordperfect, Wordstar etc. It was the Win platform change, and the huge effort MS put into using the platform change to destroy the competing products that first gave MS dominance of the office productivity software market.

      MS were aided by the attitude of some at Lotus but the story put about that Lotus held back on porting to OS/2 / Win3.x is not true. The fact that Win 3.x was a piece of junk when it came to trying to write serious business software for it was closer to the truth. I know for one big name product of the time, who beat MS hands down for years in that market segment, the port to OS/2 was very straight forward, measured in weeks, while the port to Win 3.x took many months. Because the Win 3.x platform was so unstable. At least in that case there were no deliberate attempts by MS to hobble the product. Like with Lotus and others. If you know were to look in the source code you can still find the very suspicious code in Win16 code in the Win2k source code that is out there in the wild.

      MS also spent a huge amount of money in the channels buying customers and locking out the competition. Some of it was alluded to in the Anti Trust case. It ranged from withholding MS/DOS or Win 3.x licensees from PC vendors who traditionally bundled competing products unless the competitors products were dropped to what was basically outright bribery in the wholesale channels when 5 companies accounted for 90% of all software sold retail in the US.

      As for Excel. It was a "success" because it was the only product MS had which could actually run on Win 3.x / 95 in the early days with any kind of stability. The MS/DOS Multiplan code being such a complete dogs dinner. Even by MS standards. The only reason that Excel was kind of OK was because it had to run on the Mac without crashing every 5 minutes. The very first release did that. The MacOS enforced a discipline on the code which made it almost stable. The actual code itself was horrible when dissembled. The stuff that was not intermediate code fragments was very awkward glue code to the MacOS API traps. Easily the ugliest code I ever traced through with MacsBug.

      So the reason why you have no viable commercial alternatives to Excel is because MS spent a huge amount of time and money making sure you didnt have any alternatives. Which is why their actual net profit on ever copy of Excel shipped is around 95%. Lucrative monopolies like that dont happen by accident you know.

      1. Geoffrey W

        Re: I get it

        RE: "MS spent a huge amount of time and money making sure you didnt have any alternatives"

        Aaah, I forgot the golden rule: Every situation in Tech over the past 25 years is because Micro$oft did something bad, not because someone else did something stupid.

        1. Anonymous Coward
          Anonymous Coward

          Re: I get it

          > Aaah, I forgot the golden rule: Every situation in Tech over the past 25 years is because Micro$oft did something bad, not because someone else did something stupid.

          Not quite correct.

          The actual rule is - In every single situation involving MS they will be breaking the law in every way they can but the final death blow for almost all competitors was usually self inflicted. If the competitor shows even the most basic competence, and willingness to get in a bare knuckle fight with MS - no holds barred, the competitor will often prevail.

          Lots of examples of both situations over the last forty five years.

          There is no point bringing the Marquis of Queensbury Rules to a Cage Fight. You will lose. Every time.

      2. James O'Shea

        Re: I get it

        so.. you're saying that Lotus didn't fuck up Jazz and didn't kill Modern Jazz, eh? That they didn't wait until the 1990s to try to move 1-2-3?

        Right. It's all MS's fault. Sure it is.

        1. Anonymous Coward
          Anonymous Coward

          Re: I get it

          > so.. you're saying that Lotus didn't fuck up Jazz and didn't kill Modern Jazz, eh? That they didn't wait

          > until the 1990s to try to move 1-2-3?

          > Right. It's all MS's fault. Sure it is.

          So you obviously were not there at the time. In the business. You sound like a casual user not someone in the ISV dev business trenches. Modern Jazz was a MacOS product and MacOS was less than 3% of the spreadsheet market (by revenue at the time). Late 80's. A rounding error. End of story. Talking to the Mac team at the time I never got the impression it was anything other than revenue and potential revenue return on investment. The Mac was a bad bet at the time for business productivity software revenue. All the good R.O.I segments were creative content creation and educational. And a few verticals. No one else made real money on the Mac.

          The Win 3.x port I was referring to was of a codebase that had been ported to OS/2 in 1987/1988 and I saw running on a pre-release Win 3.0 dev machine in spring 1989. At the time the product still outsold the MS product 2 to 1. There was a Win 2.0 version kicking around internally but Win 2.x was such a joke it was never ever considered for release. The guys in Cambridge, MA were doing exactly the same sort of stuff at the time with their product lines. I think their first OS/2 release came out in '89. But there again we did not have to deal with the guys in Redmond very deliberately adding code to the Win 3.0 codebase to crash our products. Much easier to do with spreadsheets.

          Yeah, there was some hubris with some of the MS competitors but if you were in the business and followed closely the Anti-Trust case an all the other MS lawsuits over the decades (there are hundreds) you would know that MS would stop at nothing to destroy anyone they perceived as even the smallest potential competitor. As Jerry Kaplan at Go and Jean Louis Gasse at BeOS (to name but two) learned the hard way as they completely underestimated the criminals depths MS would stoop to in order to destroy anyone and everyone they perceived as competition. No matter how minor or peripheral.

          MS was little better than an organized crime outfit in how it operated.

          1. James O'Shea

            Re: I get it

            Modern Jazz was going to _start_ as a Mac product, to be moved to Win 3.x. It was abandoned. 1-2-3 for Windows and Mac was very, very, VERY late. It's not MS's fault that Lotus was late. And when Win 95 and NT came out, those who had Win 3.x and/or Mac applications up and running had an advantage to getting them on the new systems, so Lotus being late to Mac and Win 3.x bit Lotus _twice_.

            And, oh... I recall there being swarms of word processors and databases and spreadsheets for Mac and Win 3.x. Excel was in the lead, but Word was not. And Access didn't exist. (Still doesn't on Macs; FileMaker Pro kicks its ass, and MS knows it and won't even try to compete.) Some of the other word processors (Nisus on Macs, for one) are still with us. Word Perfect came to Macs with lots of hype, and died a horrible death not due to MS machinations but due to massive incompetence on the parts of, successively, Word Perfect Corp, Novell, and Corel. I had a copy of Word Perfect for Mac, it did things that no other word processor, on any platform, was able to do at the time. Shame about how management screwed things up, starting with an insane marketing campaign and ending by _requiring_ users to _pay_ for certain updates (yes, plural) before being able to get the 'free' update to make certain features work with versions of Mac OS beyond 7.5.x.... and this after previous management had given those updates away... For a while WP for Mac was available for free from the vendor's site. Then they tried to charge for it. Then they just pulled it. It still worked into the early days of OS X, until Apple killed Classic, you just couldn't get updates and by that time other word processors had passed it by...

            It's simply a fact that Excel wound up ruling because there sumply wasn't any competition worthy of the name. The big vendors ignored Mac and Win 3.x, and paid for it when 95 and NT arrived. Many small vendors who went to the small ponds of Mac and Win 3 are still around, making money, selling product; Lotus no longer exists as a separate company, and neither does Borland, Word Perfect Corp, and a host of other larger operations. It's not MS who killed them, it's their management.

      3. LDS Silver badge

        "the port to OS/2 was very straight forward, measured in weeks"

        Sure, we all saw those OS/2 ports coming from everywhere.... porting a text based DOS application to any GUI OS was a task requiring a deep rewrite. If it had been just a matter of weeks, OS/2 would have had all the software it needed before Windows 3.1 had any chance to get it. Or did you mean a text-based port to OS/2? It was easier, but those had none of the advantages of GUI applications.

        Sure - having two incompatible GUI OS posed a dilemma - which one will succeed? Obviously MS had far less issue because they cared about Windows only. IBM too didn't instilled trust when it tried to make the PC architecture proprietary through the Microchannel architecture. Those who had bet on Windows early like Adobe are still here.

        Lotus didn't have OS/2 ports even after it was bought by IBM - I know because i tried to use Lotus SmartSuite instead of Office - but that was very ugly software. A lot of time spent trying to reinvent the UI instead of fixing bugs and making it useful.

        Moreover even under DOS 1-2-3 was already suffering Quattro competition, since 1-2-3 lacked even graphics and laser printing layout - both requiring expensive add-ons - in a time when without LANs deployed everywhere prints mattered far more than today. And the time and resource spent to try to block Quattro could have been used far better.

  6. gormful

    Brilliant! I nominate this for the "Single Most Useful Reg Article Ever" award.

    I'm going to share it with every last one of my users who talk about "building an Excel database".

    1. veti Silver badge

      They won't understand it.

      1. Geoffrey W

        They won't read it.

  7. Anonymous Coward
    Anonymous Coward

    So good to have Rupert back, dual wielding uzis of contempt in a righteous manner. And completely correct.

  8. jbstarbug

    Alternative?

    Whats the alternative to Excel then? Is there a simple bit of software that allows you to create tables for data classification and statistical analysis without having to know SQL or Python etc?

    My wife is using Excel for some casework analysis (how many cases were apealed, overturned etc along with various reasons). She is not very familiar with it and when I have been helping her it really strikes me just how easy it is to screw up and Excel spreadsheet/pivot table.

    1. vtcodger Silver badge

      Re: Alternative?

      "Whats the alternative to Excel then?"

      OpenOffice/LibreOffice I suppose, but they seem to me more or less Excel look alikes. Koffice, but I don't think it's maintained any more. Emacs org-mode tables, but they are not for the faint of heart. Personally, I was kind of fond of the spreadsheet in Microsoft Works, but I think getting it to run in a modern software environment might be, at the very least, challenging. And converting it's files to any other current format might be even more difficult.

      1. Yet Another Anonymous coward Silver badge

        Re: Alternative?

        AWK obviously - the answer to a text processing problem is always AWK

        1. tfewster Silver badge

          Re: Alternative?

          Really? Please tell me how to get awk to recognise a comma is "quoted" and is not a field delimiter.

          1. DarkwavePunk

            Re: Alternative?

            Cheating I know...

            echo '"foo","this is, foo"' |gawk -vFPAT='[^,]*|"[^"]*"' '{print $2}'

            "this is, foo"

      2. katrinab Silver badge
        Unhappy

        Re: Alternative?

        If your problem is "I am using Excel for something that should not be done on a spreadsheet", Libre Calc, Gnumeric and so on are not the answer. They may be better in some aspects, but they still have the fundamental problem that they are spreadsheets.

        1. bolac

          Re: Alternative?

          This is not fair. Gnumeric is actually very popular among scientists for decades because it is calculating accurately and mathematically correct.

          1. Anonymous Coward
            Anonymous Coward

            Re: Alternative?

            I remember when the Gnumeric developers found themselves in a quandary over importing Excel files. Do you support the buggy Excel behaviour, or process the data correctly and get complaints that Gnumeric isn't "Excel compatible".

            1. Anonymous South African Coward Silver badge

              Re: Alternative?

              or process the data correctly

              I will prefer that option rather than supporting bad Excel behaviour.

          2. Jan 0 Silver badge

            Re: Alternative?

            @bolac

            It is very fair. Gnumeric is just a spreadsheet. Does anybody expect it to function as a database or a word processor?.

            If you want to do statistics properly, use a Statistics package (for example Minitab).

            If you want to create graphs, use a graphing package (for example Graphpad).

            You can fill in the rest...

      3. Sanguma

        Re: Alternative?

        for KDEOffice read Calligra - Calligra Sheets is the new name for the KDE spreadsheet application.

    2. Flocke Kroes Silver badge

      Re: Alternative?

      For something approximately single user I use CSV edited with a text editor for the data and python for report generation (reportlab) and emails. There are a mixture of costs and benefits.

      The source data will probably arrive in multiple spreadsheets with fields in inconsistent columns. This can be converted to CSV an run through some python regexes to spot bad data, data in the wrong column, floating point phone numbers, duplicate records and all the usual mistakes caused by data entry into a spread sheet.

      Update requires discipline to deal with commas inside fields and adding the correct number of commas when there are a few blank fields in a row but you can recycle your initial data correction tool to report these problems. (Update turned out to be completely beyond the ability of one trying-to-be-helpful Mac user who did not have/could not find a text editor or get a word processor to output in any format usable without whatever strange software she was using. [ended up with pdftotext and a script to get to CSV]) The good news is that problems are easily visible, machine detectable and human correctable without arcane knowledge of the guts of excel.

      Next comes your first big payoff: automatically generated documents can have a consistently spelled names, addresses, emails and phone numbers for each job title. The entire document set can be automatically regenerated when any contact detail changes or when a different person takes responsibility for a job.

      Basic queries can be done with awk | wc -l.

      Mass snail mail can be handled with a PDF for the documents, a PDF with one envelope sized page per recipient and one of a number of companies that can combine the two and sort out postage.

      Python has the libraries required to send emails with attachments. It is easy to add non-trivial logic to handle who prefers snail mail, who has expressed interest, who has already paid, output a list of who would receive what so you can check you got it right and send only to yourself so you can proof the output.

      Python has excellent documentation but requires troublesome thinking skills. Reportlab has a barrier to entry but the PDF output will not display differently because someone has a different version of Word with the wrong printer driver selected. Taking the time and effort gives you valuable skills not tied to technological lock-in, spyware and adverts.

      Finally on project handover some computer illiterate can import the CSV data into Excel and think they can do what you were doing because Excel is expensive software used by business professionals.

      1. Frederic Bloggs
        Headmaster

        Re: Alternative?

        It's deeply unfashionable, I know, but could I gently point out that perl was not only designed for this sort of job, but also has modules that, for instance, can read Excel spreadsheets directly. If you insist on doing the extra 'convert to CSV/TSV|SV' or whatever step, it has a module that can read that *intelligently* as well. Perl has probably the best regex engine out of all the mainstream languages out there for processing data and transforming it. It can export the result straight into just about any database that exists. If you are mad enough, you can write the result back into new Excel spreadsheet file :-)

        Perl has an IMO undeserved reputation for being difficult to learn. But any programmer worthy of the name could learn enough to do this sort of job by reading "Learning Perl" and some judicious use of StackOverflow in a hour or three. This sort of task is a very good first practical program to write,

        I cannot stress enough that the "Practical Extraction and Report Language" was designed for these jobs - the clue is in the name. And it's faster than Python.

        1. Brewster's Angle Grinder Silver badge

          This/These

          I love Perl, but it has a huge learning curve. The @ and % determiners, slicing, typeglobs and references and a helluva lot of magic that took some getting used to.

          1. Frederic Bloggs

            Re: This/These

            Certainly if you want to do complex things efficiently, say a real time world wide distributed messaging system with ~6000 concurrent users, then you need to dig a bit deeper. Or you simply display that important attribute of a programmer "hubris" and then decide, as I did, to just start writing that app to learn perl. After all: "how hard can it be?". Answer: after understanding some subtle concepts, and using lots of examples written by other people, "not that hard at all".

            Another benefit is that this nearly 93000 line perl program (obviously split up into modules) written for linux/unix, runs on Windows with only about 20 extra lines (all to do with networking).

            But for this job. Simple basic perl and some CPAN modules is all you need. I could probably write it in less than 100 lines (assuming no extra processing other than accepting lots of little spreadsheets or CSV files, some checks and adding them to a database). You will need to know that @thing is a array and %thing is a map - but then you need to know that for any language. None of the other things you mention would be required. Definitely no magic nor subtle concepts required here.

            1. Brewster's Angle Grinder Silver badge

              Re: This/These

              Caveat: my perl is very rusty these days and out of date. So I may have screwed up the examples.

              Some of why it was hard may be how it subverted my expectations as a programmer. I spent a lot of time struggling to work out which sigil to use. "my ($x, @x); $x[0] = 4; print $x;" is a brain fuck when you're used to sigils being used to delineate the underlying type and not vary before your eyes (cf. Basic's x$ x@) And then local doesn't do what you'd expect. Maybe a newbie to programming would find it easier with a clean slate. But I found it harder than mastering C or asm.

              In terms of magic, I give you chop and all those other "operators". And you're going to need references for the DBI. It does look as if open is now vaguely sane. (i.e. bareword handles and the aforementioned local don't don't have to be mastered. And a three arg form mean you don't have to worry about magic filenames.) We're glazing over regexs but they're never far away and can render examples impenetrable and exactly what you might need to coral some ill formed CSV. In fact, this is example code from the chomp man page:

              while (<>) {

              chomp; # avoid \n on last field

              my @array = split(/:/);

              # ...

              }

              That's why I love perl, but you need a good grasp of what's going on. And a lot harder to explain than the equivalent in most other languages.

              Perl selects for people who are smart and willing to put in the effort. That's a good combination for producing good code. But I'm not sure its for a newbie and not where I would start them. And, alas, I would struggle to find an excuse to start a new project in perl these days.

    3. Loyal Commenter Silver badge

      Re: Alternative?

      Learn SQL.

      Seriously, if the concepts of "tables", "rows/records" and "columns/fields" is too onerous for your wife, then she should just give up before she starts.

      There is plenty of material on the internet on how to write SELECT, INSERT, DELETE and UPDATE statements. Basic SQL is very easy to pick up. There are also plenty of free SQL compliant databases. Yes, they might be a little tricker to install and configure than Excel (for instance the free versions of MS SQL Server, and indeed the paid for versions, aren't the friendliest to install), but half an hour spent with google should get you past any problems.

      Then she can do it properly, and if she actually needs something to be in Excel, it can be extracted from a proper database easily enough.

      Once you know how to use SQL (and have got past the inevitable mistake of forgetting the WHERE clause), you'll find there are far fewer pitfalls than there are with Excel.

      1. Geoffrey W

        Re: Alternative?

        RE: "Learn SQL"

        Don't be silly. The only way there are fewer pitfalls than Excel are if you look only at the pitfalls of Excel. SQL databases add a whole raft full of other pitfalls. And expecting a person who is barely coping with Excel (no Ms W, I am not looking at you...) to learn SQL and databases is like expecting cats to work the vacuum cleaner round the litter box; They wouldn't want to even if they could because...Why?

        1. logicalextreme Silver badge

          Re: Alternative?

          I think that you'd probably find quite a few people who use Excel inappropriately are in fact silently struggling with all the nonsense it brings, and have a head far more suited to databases. You can see the lights go on in their eyes when you explain the basics of the relational model to them. SQL is just the fairly-standard means of interaction with that model, and as a bonus it didn't get arbitrarily rewritten to be upside down and back-to-front with keywords in a different language and character set in 2007 (that's the Ribbon interface, for those playing the analogy drinking game). Learning anything never ends, but if somebody's realised that it's actually going to solve their problems, they'll learn under their own steam.

          Crucially, the pitfalls of an RDBMS don't tend to be data inaccuracy — standard SQL and the way numeric types are stored can admittedly make them a bit shit at maths sometimes, but they don't go fucking existing data up just because you looked at it funny. The fact that Excel has the concept of a "format" and no concept of data types makes it fundamentally unsuitable for anything but the most basic, unimportant data tasks.

          1. Primus Secundus Tertius

            Re: Alternative?

            @logicalextreme

            "Excel has the concept of a "format" and no concept of data types".

            Excel has major types: text, numbers, and dates. But its default is 'general', where it just analyses guesses. General is the general cause of all the generally quoted faults in this article.

            1. logicalextreme Silver badge

              Re: Alternative?

              Dates are numbers, so I think it must just have strings and numbers and then differing behaviour based on how a cell is formatted (which is utterly cracked, as a DBA).

              "general cause" :D

        2. Loyal Commenter Silver badge

          Re: Alternative?

          If you're doing any sort of data analysis, a SQL database is the appropriate tool. If you don't have a head that can understand databases, you don't have a head for understanding data, and you are doomed from the start.

          Putting your data into a database allows you to properly organise it and analyse it consistently (Excel really doesn't and has a number of pitfalls to catch the unwary). It is the right tool to be able to store the data in a structure that represents its real-world structure (rather than a flat spreadsheet).

          You might balk at the idea of using something like MS SQL Server, but if one is starting from scratch, there is probably a lot less to learn than learning to use Excel to get the job done.

          You also have the benefit of it being transactional, and logged, so you can often undo your mistakes (if you know how), unlike with Excel, where a mistake can mean your data is gone, with no idea why or how.

      2. ibmalone Silver badge

        Re: Alternative?

        SQL would not be my choice for data analysis (see reply further down). For storing, sorting, extracting data yes, you can even do some of your data cleaning and preparation in it, and I'll confess to having done the occasional avg() and group by, but the statistical tools it offers are very limited and how are you going to draw a graph?

        1. ibmalone Silver badge

          Re: Alternative?

          Though I'll modulate this to say that SQL concepts like tables and joins are transferable to other tools for manipulating datasets, it's just not necessarily where I'd start from if you've already got the data to analyse.

      3. Naselus

        Re: Alternative?

        "Learn SQL."

        Seriously, this.

        If you have a problem that requires a database as part of the solution, then I'm afraid you basically need to learn how to identify and use a dedicated database system. Attempting to make your own in a glorified calculator because you can't be bothered learning to use the correct tools is like deciding to design your own car because you can't pass a driving test - it's probably harder than just doing it properly, the result you come out with doesn't solve the actual problem (your inability to drive safely), and the fact that you don't understand how to use the existing solution suggests your own is almost certainly not going to be better than the ones real experts have been refining for 100 years.

        Learn to identify the right tools for a job, and then learn to use those tools. The reason for this track'n'trace fuckup is a complete failure on both counts.

    4. cornetman Silver badge

      Re: Alternative?

      Whatever happened to Access?

      1. Yet Another Anonymous coward Silver badge

        Re: Alternative?

        Excel is the new access!

      2. Glen 1
        Coat

        Re: Alternative?

        "Whatever happened to Access?"

        It was taken over by Mastercard.

      3. Anonymous Coward
        Anonymous Coward

        Heretical statement

        I used to be an Access evangelist - great tool for prototyping databases and a good way to set up user friendly GUIs for data entry and reporting. But over the years I've fallen out of love with her. Her habit of changing her storage formats on every version upgrade. The way anyt programming will be borked by some random update to a Windows .dll you never heard of, the dumbed down stupidity that she can't read her own old formats - let alone useful ones like .dbf. And before anyone screams, .dbf files store tabular data in ESRI shape files, which are the lingua franca of the GIS world. If I want to read tabular GIS attributes to or from Access the simplest intermediary is...fanfare......Excel!

        Anyway.- sqllite for storing stuff, R for everything else!

    5. Dan 55 Silver badge

      Re: Alternative?

      For ETL tasks, how about Python (as you already said) or R?

      I guess there's a hole in the market for a point-and-drool interface that isn't amazingly expensive (e.g. like ETI is).

      1. Glen 1
        Trollface

        Re: Alternative?

        "point-and-drool interface"

        Isn't that what VB was for?

        1. 2+2=5 Silver badge
          Joke

          Re: Alternative?

          VB users drool and point

      2. logicalextreme Silver badge

        Re: Alternative?

        Graphical ETL, in my experience, is as bad as Excel if not worse. The more fluff you put on top of things to abstract people away from the data and the reality of what they're doing, the worse the data that comes out will be. I've reserved a special place in hell for SSIS, which in its defence is technically a free gift.

    6. TRT Silver badge

      Re: Alternative?

      The alternative?

      Open Reffine? Oh and easy to use databases? FileMakerPro?

    7. TheRealRoland
      Coat

      Re: Alternative?

      MS Access?

    8. ibmalone Silver badge

      Re: Alternative?

      Stata is the most welcoming of the stats packages I've seen, simple tasks can be done through the gui, you can do an entire basic analysis without typing (and generate the commands used at the same time, so the analysis can be stored for reuse independent of data). It's not cheap though. SPSS can also be used from the gui and isn't cheap either, but they've been teaching psychology students to use it for years, so you don't need to be particularly into programming to make progress.

      R with R studio has a bit more of a learning curve, but the open source version is of course free. You can view data, graphs are plotted in a window. Yes, you'll need to learn a little R, but it cuts out much of the boilerplate you need to do analysis in a generic language. It has its own idiosyncrasies that make it a little awkward for people from a more traditional procedural background. Strongly recommended to start with the tidyverse framework (includes dplyr, ggplot2 and others) which make it much more powerful. Here's a quick start https://www.datacamp.com/community/tutorials/tidyverse-tutorial-r

      There is a learning curve with all of these, as you have to think in a more structured way (a bit like going from a word processor to latex/lyx), and excel will get you a lot of the way for simple noodling around with simple sets of numbers provided you're careful, but with a clear defined problem to work on people can often manage to get things done with these kind of tools more easily than with a general task like "learn python".

    9. macjules Silver badge
      Facepalm

      Re: Alternative?

      It all depends upon the job you need to do. For something like your wife's casework analysis then I should think Excel is ideal. If you want to run a check on 65m detailed rows of CSV data then I suspect that you might prefer to use a tool such a CSV Linter, which strangely enough HMG actually built for nationalarchives.gov.uk to check large CSV files. I wonder why they didn't use that for PHE?

      If you are a financial institution and trying to run calculations on fractional percentage trades within microseconds then something written in Python is probably going to be far more effective than Excel, or any other spreadsheet software come to think.

  9. Quenda

    What should I use instead

    Ok we should not use Excel. What should a non programmer use instead? If you are going to destroy the city you need to build something in its place. And it needs to be as easy to use, as powerful and not require a small army or programmers or consultants to maintain.

    1. Andy The Hat Silver badge

      Re: What should I use instead

      In the late 80's one of my first newbie tasks was to move a database off a Harris mini onto a PC and I used a simple DOS-based flat file database application that was easy to set up, could search by example, generate standard or custom reports with a few keystrokes, export data etc. Worked absolutely fine and ran off a floppy! That system worked really well for years until some IT department big-wig decided it needed migrating onto Access at which point complexity and lack of usability killed it totally for the users. After a while it got migrated onto Excel where, to be honest, it was horrible and difficult but it worked *for the people that needed it*.

      If I want to do the same initial migration now the words to be spoken will probably include "Access", "SQL" and "Oracle", "cloud", "hardware upgrade", "internet stability" ... And, when the punter is scared enough of the prices, horrendous learning curve, cost of books and cost of courses and realises 90% of their investment is actually for useless software components because all they want is a simple flat file database that's easy for non-IT geeks to drive, they'll end up using Excel (probably badly) as there's bugger all else and they 'know' it ...

      Perhaps, for the average punter, we need to take a step back to the database past and simplify so the punter has an alternative to Excel? But just think of the money we'd lose if people could actually do what they wanted to *well* with minimal input from consultants, support desks, training courses, books ...

    2. gormful

      Re: What should I use instead

      How about just Excel, but with a flag that turns OFF all automatic data conversion? Just edit the strings that are there; don't strip leading zeros from stock numbers or interpret gene names as dates.

      It would be AWESOME simply to have an application to edit data in CSV files that DOESN'T "re-imagine" data values whenever it bloody well feels like it.

      1. xeroks

        Re: What should I use instead

        Percentages? DON'T TALK TO ME ABOUT PERCENTAGES!

        1. Abbas

          Re: What should I use instead

          Yes, I too hate percentajes. And merged cells. And... But dates, date conversion... It is the Hell in Earth!

          1. brakepad

            Re: What should I use instead

            Friends don't let friends use merged cells.

      2. oiseau Silver badge
        Facepalm

        Re: What should I use instead

        ... simply to have an application to edit data in CSV files that DOESN'T "re-imagine" data values whenever it bloody well feels like it.

        Yes !

        Have another upvote.

        O.

      3. A Nother Handle

        Re: What should I use instead

        Notepad++?

      4. Andy The Hat Silver badge

        Re: What should I use instead

        "don't strip leading zeros from stock numbers "

        Sorry, I'm sure all modern users are aware of the stupidly obvious Lotus 1-2-3 formatting command sequence - user realises what has happened, user swears, user edits number with leading apostrophe - to prevent this? I can't believe I still have to routinely use that 'technique' ...

        1. Dave314159ggggdffsdds Silver badge

          Re: What should I use instead

          You don't. Just change the cell type. Excel hasn't actually dropped the leading zero, it just hasn't been told to display it.

          This is like complaining about all your data being stored in Variants because you haven't bothered to define a type.

          1. Andy The Hat Silver badge

            Re: What should I use instead

            Actually you are incorrect.... try entering a leading zero number into a default Excel cell and it truncates the leading zero ('cause that's what it does) - if you then change the cell format to TEXT the leading zero is not there, the entered data has been parsed as a number, truncated and AFAIA the leading zeros are unrecoverable data.

            The last thing a single user of a multi-user sheet wants to do is find that truncation happens on a single cell entry, take the sledgehammer approach of changing the entire column format (to text or whatever) to guarantee it never happens when actually it may have already done so ...

            1. Dave314159ggggdffsdds Silver badge

              Re: What should I use instead

              Tbh, it's been a while, and I can't be bothered to fire up xl and check my recollection. You might be right about the truncation.

      5. Dave314159ggggdffsdds Silver badge

        Re: What should I use instead

        You do realise Excel will do that perfectly happily, right? You just need to select the correct cell type. And possibly turn off autocorrects too, it's been a while.

    3. Doctor Syntax Silver badge

      Re: What should I use instead

      "Ok we should not use Excel. What should a non programmer use instead?"

      Email or the telephone. To ask a programmer to put together a real application.

      For a lot of purposes the spreadsheet is good enough but at the very least you need a sanity check on the results and sometimes "good enough" isn't really good enough and you have to require better. Dealing with people's health and lives is one of those times.

      1. Anonymous Coward
        Anonymous Coward

        Re: What should I use instead

        Hello - Programming department? I need a widget. What's that? I don't need a widget because we already have a EPS (Enterprise Widget Platform)? I know but it doesn't do the thing I need it do. Can you add that functionality? Change request? OK. So when can you do it? Next year? But I need it now. I have to take any request for priority to the change request committee? When does that meet? Next year!!!. OK How about if I buy a Widget from Widget's-r-us? It needs to be evaluated by IT before it goes on a corporate system? When can you do that? Next year????????. Forget it.....scrabbles around the start menu for copy of Excel.

    4. cyberdemon Silver badge
      Facepalm

      Re: What should I use instead

      I think they should be teaching GNU Octave in schools instead of Excel. Or better yet, teach "How to make a database in Python". Django is a nice starting point! Very simple to make a good database with minimal code.

      The problem isn't "what should a non-programmer use" but instead "why are there so many people who call themselves non-programmers"

      It's a bit like asking "what type of boat should a non-swimmer sail?"

    5. SGJ

      Re: What should I use instead

      What should a non programmer use instead of Excel? They should use a programmer (or learn to program).

    6. theOtherJT

      Re: What should I use instead

      The non-programmer should explain to whoever gave them this idiot assignment that they are in no way qualified for this task and suggest that said person go find someone who's actually competent to do it.

      1. Dave314159ggggdffsdds Silver badge

        Re: What should I use instead

        Sure, right. Because that's always sensible economically.

    7. ibmalone Silver badge

      Re: What should I use instead

      Ok we should not use Excel. What should a non programmer use instead?

      Are they writing macros? Are they creating calculations that depend on other calculations? Then congratulations, they're a programmer already.

      If you run out of memory slots in your calculator, need to knock up some graphs or do some quick data entry that isn't big enough to warrant setting up a database (and doesn't have horrible GDPR implications), then excel will do. If the task has gotten bigger and you've got sheets that depend on other sheets and calculations that need to be filled down and data that needs to be pasted in and a spreadsheet named quarterly_report_v2.2_2020-q3-tmp-newmacros.xls, then stop and think what you're actually trying to do. Maybe that's what should be used instead, some introspection.

    8. Naselus

      Re: What should I use instead

      It depends what you're doing. Are you doing something that a spreadsheet is actually for? Then contine using Excel. Are you doing something that a spreadsheet is not actually for? Then go and find something that is designed for the thing you're doing and stop blaming Excel for not being a word processor/database/first person shooter/Boeing 747.

      Spreadsheets originated in accounting as basically manual calculators. Computerized spreadsheets are, at their heart, automatic calculators - much like, y'know, actual desktop calculators, only generally more feature-rich. That feature richness has led to people forgetting that what they're looking at is basically still a mechanism intended to facilitate doing fairly small numbers of relatively basic 2 or 3 decimal place calculations.

      Use it as a calculator and you're generally ok (not for hyperaccurate statistical stuff, but few people outside the sciences need that and those that do know not to use Excel generally). Use it for anything else - storing vast amounts of data long-term, or just as a table-making device or what have you - and it will probably let you down. Yes, you can use Excel to code a game, or to make art, or to store large amounts of data. You can also cook your Christmas turkey with a blow torch, but it's not really the right tool for the job.

      1. TheRealRoland
        Coat

        Re: What should I use instead

        I mean, you kinda asked for it...

        https://www.youtube.com/watch?v=CSQLVCI35vI

        "But you didn't say in the use case that it needed to be edible!"

  10. mark l 2 Silver badge

    While there are times when an Excel spreadsheet is perfectly adequate for the job, it sounds like using it for contact tracing, with what has to be tens of thousands of records, if 16000 can get lost without anyone immediately noticing is clearly not one of those times when it was the right tool.

    1. cyberdemon Silver badge
      Coffee/keyboard

      CSV import?

      Has anyone asked why in gods name did they have a CSV file with 32k+ "columns" in the first place?

      CSV itself is really not optimal for "large" numbers of columns. I think this farce goes beyond simply abusing Excel. Clearly the task was given to some poor overworked underpaid clinical admin clerk, without anyone realising that they are dealing with a large scale data management problem that should have warranted the services of an experienced database engineer.

      "using a nutcracker to demolish a wall" springs to mind.

      1. Richard 12 Silver badge
        Mushroom

        Re: CSV import?

        Turns out it was *rows*, and there were using XLS (pre 2007), with the 65k limit.

        Which is of course yet another WTF!?

      2. Dave314159ggggdffsdds Silver badge

        Re: CSV import?

        They didn't have too many columns , that was just bad reporting. Too many rows makes much more sense.

        1. Brewster's Angle Grinder Silver badge

          Re: CSV import?

          The column limit on .xlsx is 16K so that initially sounded the right size. But apparently there were multiple lines per record which is why the .xls 64K row limit came into play. I hate to think what this data actually looked like or how they got it from a CSV file to multiple rows per entry. (Did every column in the CSV get turned into a row with blank rows between?)

  11. big_D Silver badge

    Its worse than that...

    When I was "young", I worked on a sales forecasting system which downloaded data from a VAX Oracle database and performed calculations in Lotus 1-2-3. The project manager wrote a quick-and-dirty prototype in 1-2-3 and presented it to the customer as a set of "working" screen mock-ups. The finished project should have had a database and use C++... Only the customer said, "no, 1-2-3 is great! All our sales people have that already, just get the prototype working!"

    I was called into the project at that stage and no amount of wailing helped. The customer was adamant. So, we expanded the 1-2-3 model. It ran in DOS, it had around 40 sheets it loaded in one after another and ran calculations... Then it started doing funny things.

    Self-modifying /-code macros didn't help (dynamic cell references, as 1-2-3 didn't have variable). It worked fine in debug mode, stepping through the thousands of lines of /-codes. But actually run it? It fell over randomly and gave the wrong results. After tearing our hair out, we actually contacted Lotus support. They asked for a copy of the spreadsheet, they got a 2MB bundle of tables (we are talking DOS here, 640KB main memory, 1MB with Himem.sys and a 40MB drive!).

    They looked at it. They wept. Their official answer was, "forget it, 1-2-3 was never designed for anything this big or complicated!"

  12. Anonymous Coward
    Anonymous Coward

    When you need something fast without the hassle of creating and hosting a database, access, permissions ,defined structures then excel is your go to albeit it should be your temporary go to. You can't email a database or chuck it on a network drive for someone to access however every one can get into a spreadsheet. In the case of track and trace they needed something quick that could work with all the various companies and individuals doing the track and trace and enable work flows from said data. Why they didn't move it from Excel is beyond me especially the amount of money thrown at it. The way I see excel is that it doesn't need to get better because it does what it's supposed to with spreadsheets/data. Now if someone comes up with an easy to use/setup database (I'm talking excel user easy and universal software wise) they might be onto a winner.

    1. aje21

      My thought was that someone mocked up an example of how it could work (PoC) using Excel and for some reason that is where things stopped...

      By why, just why, was XLS used rather than XLSX, that bit seems almost like someone breaking things on purpose!

      1. Dan 55 Silver badge

        Maybe due an ancient library for reading files somewhere that only understands XLS?

      2. Doctor Syntax Silver badge

        "for some reason that is where things stopped."

        The reason is that manglement is incapable of understanding that a PoC isn't production-ready. Or even beta. See also that in HMG beta is believed to be production-ready.

      3. Anonymous Coward
        Anonymous Coward

        Could have been a ready made template. I've seen that happen before many times. That might explain the XLS over XLSX.

      4. logicalextreme Silver badge

        I only updated my CV to .docx a couple of years ago. Office 2003 was the last Office I could use and I maintained the file format in protest at everything that's come since.

        1. Richard 12 Silver badge
          Happy

          Use PDF

          That way you can be pretty sure they see what you see, and more importantly, the idiot recruiter is less likely to add random padding.

          1. Primus Secundus Tertius

            Re: Use PDF

            @Richard12

            ... or even change things. I have had to answer some strange questions at interviews because of what agencies do.

    2. Yet Another Hierachial Anonynmous Coward

      Why they didn't move it from Excel is beyond me.....

      "In the case of track and trace they needed something quick that could work with all the various companies and individuals doing the track and trace and enable work flows from said data. Why they didn't move it from Excel is beyond me especially the amount of money thrown at it."

      Remind me again, who is head of track and trace? And her computer orientated qualifications are?

      News next week - USB stick with spreadsheet of 16000 Corona patients found in back of taxi/bus/pub......

    3. SGJ

      Maybe 'something quick' was required in the first few days of Track & Trace. Maybe. As I have seen many times in my 40 years of working in IT (first as a developer and then IT management) the 'something quick' became 'the way we do it'. There is simply no excuse for still using a spreadsheet for such an important task months later.

  13. Anonymous Coward
    Anonymous Coward

    VBA Security Security Security Security... I can't hear you!!!

    This sounds more like a FOTW than an El Reg article. Amusing, makes some interesting points, but really belongs in Bootnotes.

    All this really boils down to "A fool with a tool is still a fool".

    Excel is great at what it does - really. It has simplifed or speeded up my job no end on occasions. It has faults and limitations (some very basic and silly), but I've not used a tool that hasn't.

    And yes, an excel spreadsheet is likely more secure than a database set up by someone who has no idea what they are doing. After all, it is rather difficult to locate a file with a simple port scan...

    1. Primus Secundus Tertius

      Re: VBA Security Security Security Security... I can't hear you!!!

      "Excel is great at what it does". Yes, absolutely.

      I have rerun some calculations I did in my student days 50 years ago. Exercises that took a half day or whole day are now done in a few minutes.

    2. Joe W Silver badge

      Re: VBA Security Security Security Security... I can't hear you!!!

      Excel is for two columns, one for money in, the other for money out and comparing those. That's it. Using it for anything close to production with the amount of data and the criticality in this most recent example is at least.... questionable.

      1. Glen 1

        Re: VBA Security Security Security Security... I can't hear you!!!

        Given the propensity for JavaScript libraries dedicated to graphing and similar, do you (the el Reg readers) think the move to web-based stuff is an improvement on Excel?

        I suppose having a JS front end implies there is a proper database somewhere at the back...

    3. Doctor Syntax Silver badge

      Re: VBA Security Security Security Security... I can't hear you!!!

      "And yes, an excel spreadsheet is likely more secure than a database set up by someone who has no idea what they are doing."

      And when the spreadsheet is set up by someone with no idea what they're doing you get the mess we're encountering now.

    4. SGJ

      Re: VBA Security Security Security Security... I can't hear you!!!

      Take a look at http://www.eusprig.org/horror-stories.htm and then tell me how secure a spreadsheet is.

      1. Dave314159ggggdffsdds Silver badge

        Re: VBA Security Security Security Security... I can't hear you!!!

        That site doesnt gave very many security issues. 99% of the things listed are fat finger errors, not Excel problems.

  14. odyssey

    Blame the user not the tool

    This reminds me of Bjarne Stroustrup's observation that there are two kinds of programming languages - the ones people complain about and the ones nobody uses. Excel is a good general-purpose tool for what it's meant for. There are lots of horror stories because it's so widely used. Any other tool that was as popular would have people misusing it. Most of the horror stories listed are human error.

    Having said that, Excel is not meant to be used as a database (why MS makes Access and SQL Server). It's not MS or Excel's fault this happened, it's people stupidly importing csv files into a spreadsheet program when databases are the right tool for the job and freely available.

    1. Nevermind

      Re: Blame the user not the tool

      Yes, or you work in a big corporate with enough clout to refashion all your textual forms as Excel spreadsheets because that's the only tool you know how to use for easy layouts (why am I thinking of chimpanzees and sticks here?)...Word obviously not your cup of tea, PDF forms completely non grata...."ah excel" you go and you even password protect it. When the user offers to make it better (read usable), just say "Well, no one else has complained so no"....

      Looking at you RR, you cockwombles.

  15. katrinab Silver badge
    Unhappy

    Anyone Remember Lotus Imrov

    No not Lotus 123, of course you remember that if you are old enough.

    Lotus tried to fix the problems with 123, which are the same problems Excel has today.

    Lotus Approach was also not bad if you wanted something a bit better than a spreadsheet but not as complex as Access. People would have had that on their Smartsuite CD back before they switched to Office, but they never used it.

    Microsoft has PowerBI which seems to be trying to do the same thing. I haven't looked at it much, and probably nobody else has either.

    You can try to make better things, but people will continue to misuse spreadsheets because that's what they know.

    1. B4PJS

      Re: Anyone Remember Lotus Imrov

      PowerBI is fast overtaking Tableau as the tool of choice in Business reporting. The underlying data engine is also built into excel as powerquery and can handle 100's of millions of rows very easily in excel.

    2. Strahd Ivarius Silver badge
      Unhappy

      Re: Anyone Remember Lotus Imrov

      The road to success: a PowerBI application developped by Accenture...

    3. Krassi

      Re: Anyone Remember Lotus Imrov

      Upvote for Lotus Approach. Back in the day it helped me do simple data handling from a starting point of knowing squat about databases.

      But the main article misses big advantages of Excel. It is flexible, easy to adapt - using it is the ultimate in agile programming, do something, change it , change it again, keep on keeping on. There are lots of real world uses that don't have large data bases; that don't stretch desk-top computing power even passed through excel and VBA; projects that don't have the time or budget to bother "real" programmers; jobs that are a little different each time. Every one has excel, so different organisations can work together.

      Ironically, excel was probably being used in this case because two "proper" software products couldn't interface AT ALL.

      Anyway, Excel is approved by the National Institute of Standards and Technology ! Have a look at this great example of excel VBA

      https://pages.nist.gov/AGA8/downloads/AGA8.XLS

      1. Richard 12 Silver badge

        Re: Anyone Remember Lotus Imrov

        No, XLS was used because the receiving end was run by a complete and total idiot.

        They use Excel to merge together multiple CSV files provided by the labs, then imported said abomination into the Trace part.

        This is Dido Hardings fault, and she should be held accountable. Of course, she's a Tory donor darling and so is instead being given an even more important job to kill people with.

  16. js6898

    The same company produces an operating system where the default is to hide file extension, thus allowing any scammer to send a file called 'nothingtoworryabouthere.pdf' and hiding the last part '.exe'

  17. bolac

    All Spreadsheets Are Not Equal

    It is not fair to say that all spreadsheets are equally shit. For example Gnumeric, the one created by the Gnome folks in two weeks as a demo app, was very popular among scientists because it calculated accurately and mathematically correct. For a decade, it was also a role model for good C programming.

    1. katrinab Silver badge
      Unhappy

      Re: All Spreadsheets Are Not Equal

      It is still a spreadsheet though, and most of the problems that cause 90% of Excel spreadsheets to have errors will manifest in Gnumeric, because the problem isn't with the software itself, but with the fact that you are using a spreadsheet instead of the correct solution for your problem.

      By the way the bottom right cell in my copy of Gnumeric (1.12.46 from the FreeBSD Ports Collection) is IV65536, same as Excel 2003. So it would not be suitable for use as a contact tracing database for exactly the same reason that Excel 2003 isn't.

      1. bolac

        Re: All Spreadsheets Are Not Equal

        What does it even mean? SQLite only has 2000 columns. The idiocy is using a column for each case to begin with.

        The number of rows is 16 million.

  18. Anonymous Coward
    Anonymous Coward

    This is a job for Clippy

    "It seems you are trying to create an [X]. I think you'll find that [Y} is the right tool for the job."

    1. Aladdin Sane Silver badge

      Re: This is a job for Clippy

      "It appears you are using Excel for a database. Do you want a slap?"

  19. jo so

    30 years ago I remember an office department manager asking me, a programmer, how to improve his assistant's productivity. Every morning his assistant would log into a commercial off-the-shelf software program to run a bunch of reports to manually input into a spreadsheet to help measure sales, commissions, and goals. I analyzed the process, wrote some COBOL or RPG II program on a DEC VAX, and presented a new daily batch procedure that the assistant could run instead. My five hours of programming saved an hour each day of the assistant's time so that the assistant could spend more time on other needs. Those were the days.

  20. Doctor Syntax Silver badge

    Step back here and ask a few simple questions (the answers might not be simple).

    1. What data and in what format is received from the testing centre to accompany the samples?

    2. What format do the lab instruments export?

    3. What needs to be done to interpret the results from 2 and marry it up with 1.

    4. How many systems need to be fed from this?

    5. What format(s) are needed from that?

    6. What needs to be done to prepare the output from 3 into 5.

    7. What sort of scale is needed?

    8. What's needed to accomplish 6 at 7 reliably and at appropriate speed?

    9. What's needed to oversee the process and ensure its all running properly?

    At that point you have a set of requirements for the processing needed. "A spreadsheet jockey" isn't likely to be the answer to 8 although it might e the answer to 9.

  21. Neil Barnes Silver badge

    Forgive me for I have sinned...

    I have used excel to automatically generate the C-code for a stinkin' great state machine.

    I'm sorry. I'll try not to do it again.

    But at least it's in the revision system, so if someone buggers it up with an accidental drag'n'drop, we can at least get back to something working.

  22. c1ue

    The OP is angry at Excel.

    The OP apparently would prefer everyone use some specialized tool or language where the operations are utterly opaque.

    Excel - you can see and trace everything going on. Not so for "modern" platforms.

    Yes, it isn't scalable for Big Data or even medium Data - but it never pretended to be.

    Being angry at something which can do so much, well enough, that it is pushed to the limits and beyond is silly since this applies to literally everything in IT.

    1. Anonymous Coward
      Terminator

      Assembly language – you can see and trace everything going on.

      1. ICL1900-G3 Bronze badge

        Ahh

        ...assembly language, the only programming I ever really enjoyed.

      2. c1ue

        Oh great - now everyone needs to learn to code in assembly in order to tabulate a row or column of numbers?

        I'll pass.

    2. SGJ

      I have seen spreadsheets so opaque it was impossible to figure out what they were doing. No naming of ranges or cells, no documentation, links between Excel files, no understanding of relative and absolute addressing, columns in which all cells contain expressions - except for some 'special' cases.....

      It is possible for a competent developer to produce a good spreadsheet except a competent programmer wouldn't be using a spreadsheet in the first place.

      1. c1ue

        Except with Excel - you can stick in a query statement wherever you feel like without recompiling.

    3. logicalextreme Silver badge

      I think that it is pretending to be though. More and more and more features get thrown into it. The row limit was upped from a perfectly sane 65,536 to over a million. The column limit was upped from 256 to over sixteen thousand. Where before people would have stopped, taken a step back and asked if it was the right tool for the job (mostly…some might end up partitioning over multiple sheets before they start wondering, and there's always the odd person that can't think for themselves and never even considers that a review might be in order), there are now 17 billion cells up for grabs on just a single sheet of a workbook.

      If we can get angry at firearms manufacturers and tobacco companies for being enablers, we can sure as hell get angry at Microsoft.

  23. Peter X

    Row / column limits

    Going out on a limb here, but maybe the row/column limit on Excel should be far far far lower? E.g. 100?

    Then, people would quickly realise, it isn't the tool for the thing that they think it's for. And hopefully they'd then, correctly, use an RDBMS.

    Oh, and after a user has watched YouTube videos and (maybe) checked the help information on how to use a pivot table, when they actually try to choose Pivot Table from the menu, it should pop up a message saying "You know this is not some magical thing; it's simply grouping and aggregation which is total piss to do with an RDBMS".

    Then - perhaps - they'd know!! They'd understand!!!

    1. logicalextreme Silver badge

      Re: Row / column limits

      A thousand times yes. People's ire was directed at the Ribbon in Excel 2007, when it should have been at the limits.

      I've come to assume that Microsoft knew that they'd be able to make more from per-user Office licensing than SQL Server core licensing, pricey as it is, because previously there seemed to be a perfectly obvious progression when you hit the more obvious limits of the software.

    2. Dave314159ggggdffsdds Silver badge

      Re: Row / column limits

      Absolutely not. The 65k limit is a real problem for proper spreadsheets. The whole damn point of spreadsheets is to beat the limitations of manual tabulation.

  24. Abbas

    Rhetorical question.

    Given the proper amount of resources - money, time, skilled programmers and a Manhattan Projet like commitment - would it be possible to redeem Excel?

    1. Danny 2 Silver badge

      Re: Rhetorical question.

      Would it be possible to redeem Tetris? It does what it does, and it doesn't do this. A bad craftsman blames his tools.

  25. Anonymous Coward
    Pirate

    I don't think the problem is Excel

    I think the problem is stupidity, massive educational failure, or both.

    I mean, I'm sure Excel is adequate at the things spreadsheets are adequate at[*], the problem is people using it for deeply inappropriate tasks with no understanding of the issues around that at all. I worked somewhere where people knew not to use Excel in this way. So they stored a huge and critical system configuration in a google sheets document with no version management at all because this was, obviously, better. But not better enough: the thing was obviously too hard to use due to the whole having to be logged in to your google account thing ... so they turned that all off. It was world writable in a sense which is hard to imagine: if you had access to a reasonably modern web browser and an internet connection you could edit it, with no need for that whole annoying 'authentication' or 'having an account' thing. I should have a look sometime to see if it is still there and if my little test edits are still in it. This was less than five years ago, And yes, of course it was government-related. And of course they did fuck all about it when I told them.

    [*] Ok, I'm sure it's not. But if it was it would not help deal with the real problem, which is people.

    1. theOtherJT

      Re: I don't think the problem is Excel

      The problem here as I see it is that like a bunch of microsoft products it does a ton of things it has no business doing. The correct response should be "piss off that's not what this is for" but for some reason every time someone says "Hey, I want to do <pick a thing> with Excel" microsoft just decide that implementing the most half-assed possible way of doing that thing is the right answer as opposed to saying "No."

      See also the fact that Word is for some reason a web browser, and that Outlook does.... sort of everything that outlook does.

      1. Primus Secundus Tertius

        Re: I don't think the problem is Excel

        I find it jolly useful that MS Word can read html files and turn them into docx (or pdf or txt or even odt).

        But I agree with you about Outlook: it seems aimed at people who are pretending to be businessmen.

        Libre Office Word can even produce and read back what it calls docbook. But other xml software I have says it is well formed but not valid.

    2. Chris Miller

      Re: I don't think the problem is Excel

      A big +1 for education. 25 years ago, I was responsible for 2,000 desktops transitioning from Win 3.1 to Win95. We had three staff dedicated to training, going round the country teaching staff how to use Windows, Word, Excel etc.

      There's an assumption that you can take a work experience bod, give them a copy of Excel and they can just get on with it. And the problem is that they probably can produce (after an unnecessary amount of trial and error) something that resembles a working spreadsheet. But they won't know that if you've a table of financial figures summed by row and column (imagine sales numbers divided by product and calendar month) that you should compare the sum of the sums of rows to the sum of the sums of columns, and if they don't agree something has gone wrong. And that's what education teaches you.

      Instead we get moaning Minnies like the author of this article saying that Micro$oft is crap, when really the problem lies with management cost-cutting.

    3. Aging Hippy

      Re: I don't think the problem is Excel

      The fact that Excel is the wrong tool for the job is irrelevant here. Hasn't anyone heard of reconciliation totals. At its most primitive:

      number of tests received at centre = number of tests sent = number of tests done

      Either the person "designing" the system had no idea or, more likely, their manager thought it was a waste of time.

      Punishment should be to be taken back to the 60s and made to feed several boxes of 2000 cards into a card reader which rejects 1 in 100 while the PFY drops several handfuls onto the floor. Reliability breeds laziness.

      Excel does have a purpose here - it's presenting the reconciliation totals for all files received.

    4. c1ue

      Re: I don't think the problem is Excel

      The problem is that Microsoft's Excel programmers did a really amazing job of integrating all manner of extra capabilities into Excel.

      You can query web pages. You can move data back and forth between sheets and even documents. You can bend, fold, spindle and mutilate cells and their contents.

      That's the real problem: if there isn't a 100% perfect program to do something, it is easier to do it in Excel than to do "real" programming to create something custom.

      So the real complaint is about Microsoft Excel/Ajax programmers being too good at what they do.

  26. Danny 2 Silver badge

    CND

    I volunteered as computer support at CND in 2003. They at least has an actual database, but it wasn't reconciled and it wasn't legal. I did my best to fix that.

    The idea that in a pandemic the NHS is relying on outdated spreadsheets in 2020, I am honestly crying. WTF?

  27. Trubbs
    Stop

    Golden Goose

    I have paid a considerable portion of my mortgage supporting business critical full blown applications hosted up the back end of excel workbooks. They have just one big button to activate and everything else is VBA forms etc. The original developer left the company and as soon as I let slip I knew how to fix them I was encumbered.

    Migrating/ rewriting to 'proper' desktop apps paid some more mortgage.

    So I have a personal grudging soft spot for excel. Don't get me started on fsckng msaccess!

  28. dedmonst

    Disappointing

    It a little frustrating that el reg think excel is the story here rather than “if you starve public services of cash for 10 years don’t be surprised if their systems are held together with sellotape and string”

    1. myithingwontcharge

      Re: Disappointing

      "It a little frustrating that el reg think excel is the story here rather than “if you starve public services of cash for 10 years don’t be surprised if their systems are held together with sellotape and string”

      They claim to have spend something like £12 billion on this broken bollocks. Money is not and never was the issue, it was a total lack of skills and clue. I mean they appointed Dido Harding to run the thing FFS. If that doesn't tell you that it was never expected to work and someone's just pocketed the money, nothing will.

    2. SundogUK Silver badge

      Re: Disappointing

      The NHS has not been starved of cash - its budget has increased every year of those 10. The problem is that they spend it on useless administrators and diversity managers rather than where it's actually needed.

      1. Anonymous Coward
        Anonymous Coward

        Re: Disappointing

        Yes sure, because a millionaire Frenchman paid someone a few hundred grand to write bollocks and print it in red top rag doesn't make it true. And the irony of complaining about top heavy management in a thread about a management failure.....

    3. Dave314159ggggdffsdds Silver badge

      Re: Disappointing

      Maybe because the nonsense you state is completely counterfactual? "Austerity" was just Tory propaganda Labour were stupid enough to think would benefit them, and so didn't push back against.

      What we actually had, and still have, is a social care funding crisis. Austerity let them escape dealing with it, but it wasn't real.

      1. ButlerInstitute

        Re: Disappointing

        While we're at it, we can blame austerity on another Excel failure....

        (Probably in the article already mentioned. I heard it from Hannah Fry on Lauren Laverne's R6 programme on Tuesday).

  29. Imhotep Silver badge

    Company Standard

    "I present this list – 44 pages long – of spreadsheet horror stories"

    Uhh - could you put that in a spreadsheet?

  30. Sparkus Bronze badge

    Once Upon a Time.....

    The end-user agreement and T&Cs for Netscape, Mosaic, and first gen MSIE had requirements something like "you will not use this software to control or monitor any equipment or process that could pose a danger to human life including nuclear power plants, medical life support equipment, and military weapons systems."

    Was the subject of much lunch room laughter at the time.........

  31. Blackjack Silver badge

    I freaking hate Excel

    I would rather relearn Basic to make a small program that shows the data the way I want, haven't used Basic since the 90s, that deal with Excel unless I absolutely have to. And even then I can point the last version of Excel I used was from 2001 and bribe someone else to use the freaking new Excel for me.

  32. Anonymous Coward
    Anonymous Coward

    There was an alternative

    Lotus re-invented the spreadsheet with Improv. I bought it and it was fantastic. Unfortunately, it did things differently to 1-2-3/Excel/Visicalc so didn't take off. Much better, but different. You had to learn a new way of doing things. Formulae had real names instead of letter-number. You could have multiple dimensions and immediately change your view of your date, just by dragging a dimension tile to a different location. I loved it.

    This is the reason Excel is as it is. People are used to how spreadsheets worked in the past and don't want to learn a new product concept, even if it is a much better way of doing things. Microsoft could come up with a vastly improved spreadsheet concept, but if it works in a substantially different way to Excel, most people won't buy it. Lotus tried and got burned.

  33. bigtreeman

    The world's favourite database

    Excel

  34. Anonymous Coward
    Anonymous Coward

    Some of the items in

    The referenced 44 page article of spreadsheet errors seem a trifle unfair on spreadsheets.

    A company fouls up its budget because someone forgot to include pay rises. Isn't that like me blaming Word because my novel didn't win the Orange prize?

    1. FlamingDeath Silver badge

      Re: Some of the items in

      I’m ok with this, I don’t like microsoft anyway

  35. That 8 Bit Guy
    Joke

    Excel?

    Why not just use Access 2003?

    17k of csv? Easy peasy. And you can have excel as a front end too!

  36. StewartWhite
    Terminator

    Don't use the wrong tool for the job then blame the tool

    It's easy to blame Excel (and tbf it's got plenty of stupidity inbuilt) but this is just a highly public version of using what can be a reasonable tool for the wrong job.

    In a previous role I had the Accountancy team complain that their documents weren't as easy to type/use as everybody else's. Turns out they'd mandated that only Outlook and Excel were allowed to be used in their team so they weren't allowed to use Word.

    You can use a hammer and chisel instead of a screwdriver to undo a screw but if you do, Shirley it's you're fault and not the hammer's?

  37. ForthIsNotDead
    Stop

    A bit over the top

    I've been using Excel for decades. With VBA I might add.

    I typically use it to gather together configuration for SCADA projects and then have VBA spit out the SCADA config files (XML etc) at the click of a single button. Easy. Error-free, and the spreadsheet can be a single point of truth - at least in the initial stages before the SCADA system itself becomes the SPoT. Been doing it like that for decades.

    Over the years I've seen a lot of spreadsheets that should have been Access projects, but everyone knows Excel, and nobody wants to learn Access. That's a shame because it's a decent tool, too. They're especially powerful when used together, when Access holds the data, and Excel is used to present the data.

    Excel is a tool. If you use it for the wrong job then blame yourself or the people that are forcing you to use it for the wrong job. Don't blame the tool.

  38. jim.warwick

    Was this Excel or just Excel file format...?

    What I read here is that the data clean up process used an out of date Excel *file format*.

    This to me sounds like it could be a data clean-up tool that is using an old excel file format as it's output to be imported into the national database thingamyjig (whatever that is).

    Working off a deprecated Excel file spec as your data interchange mechanism is not much better, but it's at least not using some hacked-together Excel macros...?

    Would like to know the details...

  39. Paddy

    Shiny Shiny

    If a company porposefully bakes its known "1900 is a leap year" bug into its Office open XML "standard" , then shame on those who ratified the standard and shame on those who entrust their data to a company who flagrantly ignores its bugs rather than fixing them.

    Spreadsheets are as attractive as sirens; but more perilous!

  40. DenonDJ DN-2500F

    To quote Sylvester McCoy as Dr Who.

    A bad workman blames his fools

    In order of foolishness

    #excel

    #intern

    #developer

    #BA

    #Harding = #SlapHandcockMP

  41. Anonymous South African Coward Silver badge
    Coat

    Lotus 1-2-3 or SuperCalc5 an option?

    Just run it in a DOS Box...

    I'll take my coat and exit to a less complicated age, kthanxbai

  42. Caesarius
    Linux

    We need a route to using better tools. When a friend wants a pc setup, we could install Ubuntu and see if they notice the difference. When a project starts at work, specify the tools for the project. Nudge towards better things.

    But this may be hopelessly slow. Any other ideas?

    1. EnviableOne Silver badge

      Tried it https://github.com/NHSbuntu/nhsbuntu

  43. BitGin
    Facepalm

    Is anyone prepared to defend Excel for...

    hitting its limit on the number of rows and then failing silently?

    Seems like the most basic thing a bit of software can do is warn the user when it hits a hard limit like this.

    1. Dave314159ggggdffsdds Silver badge

      Re: Is anyone prepared to defend Excel for...

      It definitely warns unless the warnings are deliberately suppressed.

      1. Anonymous Coward
        Anonymous Coward

        Re: Is anyone prepared to defend Excel for...

        I even went in and bunged 70k rows into an Excel sheet, than tried saving it as .xls . The warning that data would be lost was pretty bold, front and centre.

  44. Screwed

    My most frequent use of Excel is as a simple calculator.

    Bung the numbers into cells and add a few formulas. Makes it so much easier to check for and correct typos. And, if needed, extend the arithmetic to a few rows.

    But this is largely making up for the problems of calculators.

  45. Binraider Silver badge

    Excel is, what it is, and personally I don't blame the program for it's known limitations and difficulties when mis-used.

    The bigger problem is corporate red tape associated with getting something better - personnel training, the myriad choices, ongoing support and so forth. There are no shortage of suitable tools; but cost involved in actually putting them on the ground for anything more than isolated instances just doesn't fly.

    Cross compatibility of stuff is the key to getting out of the isolated instances problem, and SQL did, for better or worse, try to create a route to that for relational data. If one could create cross compatibility of expressing analytical relationships, inside a computer system, you would be onto something. Oh wait, that's called a programming language.

    And once again, we are back to the skills / cost of deployment issue.

    This headache is not going away, until we can instruct computers to build things of their own accord ala Star Trek.

  46. Andy Denton

    The problem is....

    ...even if Microsoft were to come up with a spangly new replacement for Excel that was perfect in every conceivable way, people wouldn't use it because It's Not Excel and Excel is all these people have been used to using for decades. Many people (accountants etc) that use Excel a lot aren't adventurous types and there's no way they'd trust their precious data to some new spangly product when Excel is still available.

  47. Anonymous Coward
    Anonymous Coward

    Wrong blame

    Blame the tools who use the wrong tool, not the tool being used.

    Excel is more complicated than a simple spreadsheet needs to be and I've never met anyone using a significant proportion of its capability. But, in the right hands, it is a good tool. The problem is that it's in the wrong hands.

    I first encountered spreadsheets with Lotus Symphony and had to learn its quirks as it was being used as a database central to a multi million $ project. It had its limitations and these were accounted for - it was only pushed as far as it could be relied upon. Excel is far more capable but, as others have said, it's not a multi-user database. If I had to highlight one fault with Excel, above any other, it's that it's too easy to get out of your depth without realising it.

    1. MachDiamond Silver badge

      Re: Wrong blame

      "Excel is more complicated than a simple spreadsheet needs to be and I've never met anyone using a significant proportion of its capability."

      I've found it to be more complicated than it needs to be and then falls on it's face when trying to do really complicated stuff. Maybe it's awesome for accounts and lobotomized MBA's, but it's a PIA for engineering. The only reason I use it at all is compatibility or it wouldn't be loaded on my computer. For the places it doesn't work, I output a PDF from something else to communicate data to somebody else. I rather like that they can't mess with it.

  48. Dixx

    Data with a DIY interface (no text)

    Why should Britain Tremble?

  49. DocNo

    Lotus Improv

    It's too bad Lotus Improv never really took off - that was some amazing code.

  50. L'Ecossais

    Compatibility mode

    It seems to me that the cause of the problem here is very simple. Excel can in some circumstances default to "compatibility mode" which has XLS as the default file save. To resolve it, you have to set the default file save type in File/Options.

    As to the error message when a Save is attempted, it is likely that the import and save were automated and any error message potentially logged. As the situation came to light on a Monday, what are the odds someone came in to work, viewed the logs, or even got an automated email, and had the Oh Sh!t moment.

  51. Torchy

    Microsoft Money.

    My Wife used Microsoft Money (the version that came with DOS 6.2) in her business up until two years ago when her accountant told her that she had to use Sage.

    She could get a turnover of £720,000 accounted for down to the last ten pence.

    That Sage software is so inflexible that the accountant is happy if it is to within £100.

    When it is not broken do not fix it.

  52. Anonymous Coward
    Anonymous Coward

    Excel is Hell, but my IT support was downsized & laid off years ago!

    Don’t rain on our parade. We are stuck providing intelligence to the unknowing using data dumped into spreadsheets by that one guy left in IT. I always start my conversation to those leaders with “this shouldn’t be done in Excel”, but they are more interested in the color used in the final Excel crunched charts than long term data accuracy and maintainability. Always always always when the day comes the Excel craziness over colors drives me to leave for another, sadly similar, position - they are surprised that the younger (cheaper) version of me can’t get the chart to work - let alone layer in that shade of vermilion.

    1. Anonymous Coward
      Anonymous Coward

      Re: Excel is Hell, but my IT support was downsized & laid off years ago!

      This one post anonymous... I work with a team that at one point in the past (and may still for all I know) used colour formatting of cells as a way of storing information. Not conditional formatting, regular formatting. Red category A, green category B, that kind of thing. Apparently a lot of discussion went into the colours to use.

  53. MachDiamond Silver badge

    Not for large data sets

    Excel bites for large data sets. We had problems using it to analyze telemetry data at a small aerospace firm that we'd get back from the rockets. I found something else that worked much better, but I'm blasted if I can remember the name of it. I've been out of that company for years now and good riddance.

    If it's the only tool in your box, that's what you use. I'm the sort of person that looks for alternatives to the alternatives. I'm always working on unique projects that mainstream software companies have never considered except for one that was in the same position and wrote something custom and put it up for others to buy or use.

    I also try to test my methods several ways to make sure my results are accurate. They® don't teach that in schools. One usually learns when they get their XXX handed to them by their supervisor that could just look at the numbers and know there was something wrong.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021