Python tops programming love list – but if you want a job, learn SQL Once again, Python is at the top of the IEEE's annual survey of popular programming languages – seemingly decided by a grab bag of metrics – while SQL appears to be a crucial skill. When it comes to popular languages, the survey found Python topped the list, followed by three flavors of C (original, ++ and #), Java, SQL, and …
They still are… but at some point the lack performance of the active record pattern drives people to looking for a better way working with aggregates. Turns out, relational databases have all kinds of tricks that can save the day.
I still don't like SQL very much as a language and I do rejoice every time I work out how to do something with it "better".
A couple of months ago I came across a very interesting Postgres talk I Didn't Know Postgres Could Do That. Says perhaps a lot about my level of understanding but I'm sure many of us have had similar experiences.
I run a website for square dancing. It currently has over 8,000 pages. To control this mess I put everything in an SQLite database and have a program that reads out the appropriate data and constructs the website. To add pages I edit the database with a Java tool I wrote and remake the website. Uploading the changed files completes the operation.
The database is currently 4 MBytes.
> Seems like only yesterday when all the cool kids were banging on about NoSQL.
That was only the Mongo DB marketing department putting out press releases and creating hype.
When I first heard about "noSQL" the very first thing I did was go on Google scholar and try to find its seminal paper (the equivalent to Codd's “A Relational Model of Data for Large Shared Data Banks”).
Found nothing of the sort at all so I said to myself “this can be safely ignored”.
Honestly, I wish some journalist somewhere had taken the trouble to do that rather than simply repeating press releases almost word for word.
Still use Fortran quite often unfortunately. Completely with you on Scheme. I make every excuse to use it where I can. It's such a beautiful language. Less is more.
Rarely is there a day I don't have to deal with a problem that just wouldn't have existed if the whole thing was written in a lisp to begin with. It makes me wonder where we'd be today if it was the standard language everybody used.
There's nothing wrong with stored procedure per se. It's just that like any tool, it can be misused.
If you're dumping a bunch of complex business logic into your stored procedures, or writing a 1000 line SP with some absurd multi-level recursive query, you're doing something very wrong.
Oh yes. Those rules should be written in stone.
My all time favourite example of an infraction was a monolith of something over 2,500 lines which never once accessed a database.
A close second was a somewhat smaller example which, at one point, shelled out and.....ran another stored procedure on the same database(!)
stored procedures have their use but if you cannot change SQL engines BECAUSE of them, you should re-consider your current "locked in" status. This goes QUADRUPLE for MS SQL Server.
(my preference, PostgreSQL, as clean as possible)
But yeah, stored procedures dependent on proprietary features, WORST DECISION In My Bombastic Opinion.
Sometimes the problem is really the table and relationship design, and the initial query that gets your results. And a little bit of sane client side coding to do the things that are grossly inefficient when left up to a stored procedure... been there before. had to make that argument and then PROVE it.
Aha! Square pegs and round holes. I remember once poring over a diagram of all the software bits 'n pieces with a consultant who was trying to understand it (at least they made the effort, a big plus in my book).
At one point he pointed to a small, acronym-labelled box between two systems and said: "What's that do?".
"That is a piece of software with a square peg on that end and a round hole on that end. I know it's daft, but it was either that or build Mallet 1.0".
My experience of using other people's python is that it's almost universally crap, with the few exceptions being where it's deeply embedded in a larger package. It may be easy to write, but it's difficult even for machines to read.
Python learned nothing from the 2.7 -> 3.0 debacle. In fact, the compatibility situation got worse, with multiple step releases within 3.x making scripts written for one version unusable on another and with incomplete dependencies that make it very difficult to find the problem.
Forth, and to some extent Perl used to be decried as write-only languages. Python is rapidly outdoing them.
What's the solution ? Not venv; like any containers, they're a bodge to overcome the compatibility problems that shouldn't exist in the first place (and don't, in a well-regulated language like C). But with inaccurate dependencies, they don't solve the problem either - and in any case, need to be automatic rather than an optional extra if they're to make it usefully portable.
I've only just started looking in to python for a project I'm working on after years of doing PHP - python seems to be incredibly easy to write and make mistakes in. There is nothing forcing you to think about what you're writing, it can just be a stream of consciousness that you bang out and forget about.
And I am well aware of the irony of coming from PHP and moaning about that, but I think it's worth noting that the virtue of it's easy to write nature is never tempered (in the book and resources I've seen) with the idea that you must think through what you're writing so that it doesn't end up a mangled mess of garbage.
I think you have correctly identified the problem: Python is OK as a modern replacement for BASIC.
Unfortunately at some point the masses started to over hype it and people started assuming that somehow Python must be superior to anything else.
Yes, it's relatively easy to FFI C libraries into it and use it as a quick and dirty prototyping or one-off scripting language. Go beyond that and see your tech debt raise in a way that makes inflation look tame.
That's usually the way: when someone has decided that a substantial part of their future livelihood depends on something, they're apt to react strongly to the suggestion that it's a pile of obsolescent shite that any right-thinking employer would ban from the premises.
(And any mention of "downsides" will be mentally parsed into "it's a pile of obsolescent shite etc.")
My machine apparently has no problems reading it. And yes, the string/byte thing is a pain, but that's the only issue I've had with 2.x -> 3.x
The enforced indentation is the main reason I use it, actually. Especially after 15 years of literally randomly indented PL/SQL written by complete hacks.
Yeah I don't really quite understand why they get so worked up about indentation.
If I'm working on someone else's code and don't like the way it's indented, I just go Ctrl+I and have it my way. What's the big deal?
Besides, even in Python indentation is a problem, like when people use spaces instead of tabs (again, Ctrl+I fixes it) or when you mistakenly mix tabs and spaces or when your text editor forgets to strip trailing spaces and you end up with not quite blank lines in a block that are not indented the same as the surrounding code.
I don't blame the author for trying but they should get off their high horse and introduce delimited blocks. If nothing else, that way they could then have proper lambdas.
I totally agree with your sentiments after a week of struggling with Python versions. My prof codes in Python and I am investigating the consequences of a possible upgrade from Mint 20.3 to Mint 21 (equivalent to a jump from Ubuntu 20.4 to 22.4).
The thing is that the new Mint (and Ubuntu) now defaults to Python 3.10, which breaks most IDEs like Spyder and Eric, which require v3.9. The growing dependence of the apps and the OS itself on Python means that it is almost certain that several things will break when either updating the apps or upgrading the OS. At about the end of the OS's life cycle most of the bugs (depending on users' requests) will be ironed out, but the next OS iteration will adopt a new version of Python etc. etc. ad infinitum.
It seems that virtual environments like venv or Anaconda may provide for a temporary solution but a) they have a steep learning curve and b) seem to require a different environment for each app (at least that is my impression after several unsuccessful attempts to have both Spyder and CERN Root run in the same Conda virtual environment).
I am beginning to suspect that Canonical predicted exactly this situation a long time ago and thus chose to integrate the snap store within Ubuntu to ensure all apps are containerized together with their appropriate dependencies versions. Just because an app appears in Ubuntu's or Mint's software center does not guarantee it will run - the list is not curated (at least the two IDEs I mentioned above don't).
Time and the desire of staying up-to-date are the real enemy here, and therefore the universal rules that apply are "do not mess with a system that works", along with "test thoroughly any changes/upgrades within a virtual machine first". Thank you very much, for the foreseeable future we will stick with Mint 20.3 and Python 3.19 - everything works.
The most important thing is stop keep system libraries separate from user ones. This was solved decades ago in BSD.
As for the rest: Unix is full of this envvar manipulation precisely because it's so cheap and useful. Think of the use of different users for processes, or the fairly common process of running distinct websites in the same Apache process. Then again, BSD jails have also been around for decades.
> not a fundamental flaw in the way the Python ecosystem works.
The fact that those wrappers around bash source (and equivalent in other shells) exist at all is *precisely* because of the fundamental flaw in Python that dependencies were not thought through properly to start with.
Btw, this is what pisses me off about this python shit: all the bloody fanboys queuing up to put lipstick on the pig. Use it for what it is useful if you must, but please stop deluding yourselves or trying to convince others that Python is anything more than a half cooked hack that's not very good at anything in particular but lets you put together scripts quickly and without requiring much skill on your part.
Python's packaging was never brilliant but then, back in the early 1990s, there was virtually nothing around that supported such a modular development model.
Venv's don't copy, they link. This makes them very lightweight and flexible: most importantly libraries installed in the venv won't make it into the source/system which solves lots of problems that people used to see. Used together with a good ports system and you can easily develop and test multiple projects with multiple Python versions locally.
That said, I know Python developers who eschew the approach and insist on a clean Python install for each project. Of course, nowadays some people prefer individual guest OSes or containers per project.
Excuse my naivite...
Wouldn't it be simpler for new compilers/interpreters to maintain backward compatibility with old versions? Then require each program to declare in its header which version it's written for, and therefore which set of rules should be applied when parsing it.
That's what HTML did until v5, and it kinda-sorta worked.
It seems that virtual environments like venv or Anaconda may provide for a temporary solution but a) they have a steep learning curve and b) seem to require a different environment for each app (at least that is my impression after several unsuccessful attempts to have both Spyder and CERN Root run in the same Conda virtual environment).
You should absolutely use separate environments for each app. That's a good practice that will avoid you many problems and wasted time in the long term.
The learning curve isn't steep. Once you get around creating environments you won't even think about it. To spin up a new environment is usually down to a single command and if you have your shell configured to show you your command history, you won't even have to remember it beyond first few letters.
> It seems that virtual environments like venv or Anaconda may provide for a temporary solution but a) they have a steep learning curve
It's not that it's difficult, it's just the Cult of Python and their penchant for obfuscating things. This is how they define a "virtual environment":
"A cooperatively isolated runtime environment that allows Python users and applications to install and upgrade Python distribution packages without interfering with the behaviour of other Python applications running on the same system."
In reality, it's just a directory with a copy of the Python runtime and whatever libraries you care to put in there. "Activating" the environment consists of sourcing a bunch of environment variables into your current shell.
If this was explained like this, without all the esoteric bollocks, that "steep curve" would be just a small ramp.
I upvoted your comment, but:
can someone please explain to me what the bleep you are plotting on your "learning curve" graphs that can make
> "steep curve" would be just a small ramp
a meaningful statement in the way it is used here?!
(It sure aint "amount learnt" on the y-axis against time on the x, or "ability to do task" against time)
> can someone please explain to me
Presumably, people engaging with Python know what a directory and environment variables are, and might even have a passing acquaintance with the concept of sourcing shell instructions from a file, whereas I doubt even the person who wrote it knew what on Earth a "cooperatively isolated runtime environment" was supposed to mean.
Thank you for your rant.
I've been working with Python for over twenty years and, while the move from 2 to 3 was indeed poorly handled, lessons were learned and code since 3.3 has been forwards compatible (and backwards compatible with 2.7). The faster release schedule is a bit of a pain, and there has often been a tendency to include some trendy stuff that ends up being very niche, but, given the pace of development that's only to be expected.
When it comes to poor code, I most commonly come across this in code from programmers using idioms from other languages. I guess that this is common in other languages.
People with little or no programming experience, who are nonetheless using Python as part of their job, tend to write code that is readable even if it's not the best.
> What's the solution ? Not venv;
Definitely not! It's about the most brain dead hack one could possibly imagine.
For the unaware: python "virtual environments" consist of copying the python runtime and a bunch of dependencies into a directory and modifying a bunch of environment variables to point to it rather than to the global locations. There's nothing virtual about it so I don't know where the name comes from.
If you're familiar with NodeJS, think node_modules but out of tree.
It's important to note that there's a scalability factor for programming languages. People start with Python because it has very little ramp-up overhead. It's great for small projects. As it grows, it hits scalability limits common to dynamically typed languages. It's harder to find implementations, references, and mismatched object fields.
This is where statically typed languages win. You may curse the build and dependency tools but refactoring is a few mouse clicks no matter how big the code has grown.
Well, even Brian Kernighan mentions Python as a go-to (no pun intended) language in the video linked yesterday. (https://www.theregister.com/2022/08/23/universal_unix_tool_awk_gets/)
But then SQL pays my bills. I'm STILL doing Oracle Forms & Reports. (yes, ugh, but they pay hand over fist for someone to keep it going, and yes, I'm working on migrating away)
(well at 56, nearly)
On paper I am an IT manager for infrastructure and e-commerce. Which I am. However SQL skills (20+ years in the honing) have delighted my boss (CEO) as it meant sadly we could let 2 business intelligence "developers" go. Yes they may have been whizzes at Tableau and Metabase (they probably were). But when a 1 minute SQL snippet can deliver what a days work was needed to do (plus the ability to write back to databases to fix them) it was a no brainer.
MS-SQL, Oracle and MySql are all in a days work.
"Hey, we're starting to use this new business tool called PowerBI. It'll allow us to visualise and interpret all this data we've got. We need you to go on a 2 day training course and then we'll buy you a license. And a Windows PC to run it on."
It looks like Excel with a suit on...10 minutes of SQL later....
"You mean like this?"
"Oh."
Those were the days. Spending time with MSSQL Management Studio crunching the data.
The request comes to delete some orders from production database that developers created by accident and it is messing reports.
Ok then:
DELETE FROM orders
WHERE customer_id IN (...)
Probably you see where this is going. Someone shouts from across the office "When are you going to delete this, we need to run the report!" while someone on the left brags about the girls he took home on last night out.
"It's coming!"
Then as luck would have it I didn't notice I only selected the first line and executed the query.
The whole orders production table have been deleted.
Now I know to always write:
DELETE FROM orders WHERE
So that the line won't execute by accident.
They didn't get to create a report that day. I had to spend the night rebuilding the orders table from a backup and payment provider data and logs for the orders that didn't make it to backup.
Fun times.
> I only selected the first line
What happened to the semicolon? Did you actually type one for good measure or were you running some kind of IDE that thought it would be helpful to insert one?
I do tend to use transactions and prefix dangerous commands with -- while I'm typing, mind.
Javascript is a programming language? I thought it was a big pile of crap.
BTW, HTML is not a programming language. Even if it's awesome.
"I'm learning Python!", is just something that idiotic IT sales people people say to try and impress techies at parties.
It has replaced "I'm a whizz at microsoft dot net" and "years ago, I used to debug unix crash dumps" as the buzzword bullshit they know nobody will ever call them out on.
I've found that knowledge of SQL seems to serve as a marker for how skilled a developer is.
Basic SQL is comically easy to understand. Even if you struggle with cross products and whatnot, a basic join is trivial.
So now if someone says, for example, that they use mongodb cause sql is too hard, they immediately go into my "This person shouldn't be allowed near a computer" group and go on full defence because it's all but guaranteed that their code is going to be absolute crap. If they haven't taken the time to even try learning SQL, they most certainly haven't taken the time to learn other critical aspects of software best practices either.
I have yet to come across one single person that hasn't fit that pattern.
The same person that does "SELECT * on $TABLE_WITH_MILLIONS_OF_ROWS" without a WHERE clause, is the same person that will do idiotic things like take a user's input and inject it into an INSERT statement without even trying to sanitize or escape it first.
> I've found that knowledge of SQL seems to serve as a marker for how skilled a developer is.
Now that you mention, that's my experience as well.
> The same person that does "SELECT * on $TABLE_WITH_MILLIONS_OF_ROWS" without a WHERE clause
I'd be more worried about the absence of an ORDER BY (in a purely relational database).
Is it just me or does anyone else out there also think that SQL and CSS are not real programming languages but rather syntax for glue layers that move data in and out of databases and put style up onto web pages.
If a job requires guru-level and/or vendor-specific SQL skills then something must have gone terribly wrong in the data model design. Which can be good news for SQL gurus of course, but bad news for maintainability.
Olli, I'm sure it's not just you. SQL may not be a general purpose programming language in the way that C is, but it is Turing Complete and at the end of the day all languages move bits. It's just that SQL is geared to moving lots of them at the same time.
One place I worked was a big user of SQLAlchemy. Sometimes, there's be a query required that I knew was SQL-able.
So I, the greybeard (virtually, no beard), would make the query work in SQL, then work out how to represent that in SQLAlchemy. Which can be a pig.
SQLa can be very useful. I used to detest it along with all the other ORMs but one of the most useful things is it will let you pass in a SQL statement to run. This returns standard SQLa objects which means it works well with Pandas. It also has instrospection means you can keep the DDL out of it.
That's all well and good until who know something plus a bit of SQL find out the hard way about performance problems in production (and only in production).
Then the SQL Gods demand days, weeks, or even months of sacrifice before they are satisfied that you have left your specialist language which you quite liked programming in behind and are fully devoted to SQL, only then will they deign to allow it to work again.
SQL is supposed to be a language where you don't need to bother with mere details about how you want something doing, you just describe what you want, and the database will do it. This works until one day the query blows the database's tiny mind, and now you have to hold the database by the hand and describe how it's done using a language which is not designed for it.
Let this be a warning. Don't dabble in SQL, it's more trouble than it's worth.
> SQL is supposed to be a language where you don't need to bother with mere details about how you want something doing
You need to have a chat with whoever taught you that.
From a practical point of view (as opposed to a CJ Dateian point of view in which SQL really is a terrible fit for the relational model), any decent teacher or book will insist that you are conversant with ANALYZE before you go too far down the SQL road.
You need to have a chat with whoever taught you that.
SQL is classified as a domain-specific declarative language. You describe what you want, the database works out how it is done by coming up with an execution plan then executes it. This is CS-101.
I would be more concerned if there were someone who wasn't teaching that.
The theory is that you don't need to bother about the details, the reality is no-one has yet built a perfect optimizer. I'm not sure the perfect SQL optimizer can be built. If you think relational and SQL is bad try programming with a hierarchial (IMS) or perish the thought network (IDMS) database. Arrgh!
Well.....a long time ago there was a theory about avoiding "crap". The theory suggested that software creation would benefit from a process:
(1) Find out what the business requirements were (i.e. what did the end users want to do with the software)
(2) Examine the existing software environment and see how the requirements might fit (sometimes the word "architecture" would crop up)
(3) Develop a design, usually with some early ideas about tools (sometimes ideas like "modularity" would be articulated)
(4) It might be that items #2 and #3 needed some cycling round......
(5) .....and then programming could start, with a little confidence that the result would not be "crap".
Then, maybe fifteen years ago, we got to the notion of "agile".....which often implied that:
(6) Item #1 could be addressed as yellow stickies with a word or a phrase ("user stories") and...
(7) ....some yellow stickies were chosen at random...
(8) ....and item #5 started right away......but only for a two week sprint.
So here we are in 2022, and many of the El Reg commentards blame one or more of: people, tools (Python, SQL), fashion (SQL or NoSQL).......when all along the real problem is PROCESS.....or lack of it!!! Oh dear!!!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
