Re: I don't why she swallowed a fly
"We do now have the tools that give users more expressive power, I'm thinking here of the combination of Jupyter notebooks and Pandas."
In my experience, these create the same problems that Excel has, just in a slightly different form and only for programmers. That's the case because nonprogrammers generally don't know how to use Jupyter, and even if you can teach them, they'll only learn to run existing ones. But for programmers, effectively you've given them a way for anyone to have their quick and dirty scripts. All of us have some, and they often have some deficiencies like no documentation or a particular input format without which they won't work. This is why all my temporary scripts are on my computer and my backup directory, but not anywhere where my colleagues can get them. If I'm releasing them for anyone to use, they need to be higher quality. If other people need to run them, then I'd better get started on that.
In the one environment where Jupyter was used as a production tool, we had lots of scripts that were supposed to be run. One, for example, would take an input XML file and populate a database. As long as you read the code first, you would be fine. If you didn't and uploaded a file you didn't want to, it would not fail well. It was not idempotent and would cheerfully clobber the database if you uploaded the same file multiple times. It did not do validity checking on most of the data in there. It didn't even preread the file to make sure it all worked before it started firing off SQL statements, and it didn't even have the courtesy to do this stuff in a single transaction that could be rolled back. In short, it had all the same fragility of spreadsheets but in the form of code. A problem with the script or an edit made by someone who shouldn't have been editing could easily have broken things.
While the actual failures weren't so catastrophic, I do remember the time when I ran an analysis script which returned spurious data, only to be told that I shouldn't have run the 2.0 script, I should be running the 1.5 script (nothing said this, so I had gone with the latest from version numbers). Then, when the 1.5 script also produced incorrect results, they mentioned that I had to run a script in a different folder, also labeled as 1.5, which had been forked months ago and was now the canonical one. To be clear, the scripts I was running were in a directory called "analysis", and the script I was supposed to run was under a directory named something like "dev/colleague_username/temp". The colleague in question had left the company. Is this the fault of Jupyter's products? Clearly, no. It was the fault of the process that made running a script like that the expected procedure. The remedy is the same, though. The process, not the tool, is the problem. Changing the tool is probably necessary, but if the process didn't change, the problem probably will stay around afterward.