Fed up with Python setup and packaging? Try a shot of Rye The recent debut of a Python tool called Rye has raised hopes that the programming language's long standing setup and packaging shortcomings may be solvable. Rye, a personal project developed by Armin Ronacher, an open source developer and creator of the Python web framework Flask, addresses two issues: bootstrapping (setup) …
They work if:
* You want to use the version of Python that is installed by default on your system
* You don't want to package your work
As soon as you want to use a different version of Python for a project, or you want to publish your work on PyPI or distribute it in wheel form, there is a whole world of pain that venv and pip don't help you with.
That said, rye seems to still have some pretty rough edges. After an hour of faffing around, I still can't get it to install pylint into a venv...
They work but only provide the basics in terms of convenience. For example, if you create a venv you still have to remember to activate it, either through an IDE, or on the command line (for which I use direnv. The functionality of pip depends on whether you have: `pip install numpy` will do different things depending. And, pip is not a project tool. If you copy my git repo, you need to create a venv and then run pip on requirements.txt. And, because there is no lock file, what you get might still be different from what I get. And if I have run `pip install numpy` previously, my repo might not work for you because I forgot to add it to requirements.txt.
And so on. pip and venv are potentially part of the solution, providing installation, dependency management and sandboxing. But they fall wall short in providing all of the solution. Python packaging is a mess.
I can't figure out what was wrong with this. I blew it all away and started again and it all just worked. It had somehow got it into its head that it needed quite an old version of pylint in that one project and nothing I did seemed able to convince it otherwise.
I mean, I've never really had *that* much trouble. This is what the packaging/release process for all my Python projects looks like:
https://github.com/os-autoinst/openQA-python-client/blob/main/release.sh
(there are some simpler ways to do the version stuff these days, but that's what I'm used to). Run that script, release is done (published to pypi). I could ditch setup.py and just use pyproject.toml , but that'd maroon far too many older setups, so I keep it around.
Local testing via tox works fine: https://github.com/os-autoinst/openQA-python-client/blob/main/tox.ini (the isolated_build setting for tox makes it not use host system packages, which reduces any chance of different results on different systems), just needs the interpreters for whichever Python versions you want to test installed, which Fedora makes easy. CI just runs tox in containers: https://github.com/os-autoinst/openQA-python-client/blob/main/.github/workflows/tox.yml
However, there's so much noise about this, it must be a real problem for some cases, I've just always had trouble grokking exactly how. I suspect it's partly much more of a problem if you don't develop on Linux (note all the discussion of getting Python installed in the first place, which is never going to be a problem on Linux...), and partly much more of a problem if you don't work in pure Python (so compilation is involved at some point)...
I find Hynek Schlawak a really good source to keep up to date on stuff - https://hynek.me/articles/ .
Have an upvote.... in fact the only upvote for some reason.
I'm with you on this. Nothing wrong with a requirements.txt, a supporting .md and a small directory with your systemd service files and startup scripts in.
That said, Armin is a huge source of talent and Rye has just gone into my priority learning list. I'm still waiting for him to convince me asyncio is better than Gevent though.... or anybody for that matter.
> bootstrapping (setup) and packaging
An interpreter doesn't need "bootstrapping" outside of compiling the thing in the first place. Anyone who thinks it does is making things difficult for themselves.
And by "packaging" what they actually mean is encouraging a culture of dragging in a load of technical debt and inadequate "square" wheels as dependencies.
NPM, CPAN, crates.io, PIP, etc. No thanks. Short term convenience, rarely outweighs the long term maintainability.
(not to undermine the work involved however. Orchestration of all these systems is very impressive, and quite a feat to maintain).
The same way I install packages for any software...
Sorry, I can't quite work out why you are asking such a seemingly elementary question. How do you install software in general? How do you think Python developers achieved anything before pip? How do all the other hundreds of languages who don't provide a language based package manager get around the issue?
So many questions, but honestly none of them are something that someone working with IT should be asking.
> How do you think Python developers achieved anything before pip?
Well, I can only speak for myself, but, depending on the level of inconvenience and self-torture I was looking for, either by installing modules via the distros pkg manager, or by downloading them in whatever format the maintainer thought was most useful, and shoving them somewhere into `sys.path`.
And both methods sucked. Hard. That's why first easyinstall and then pip where invented, that's why it became the dominant tool for installing modules. That's why almost every single major programming language other than C/C++ have at least SOME concept of a built in pkg manager.
Distro pkg manager sucks because it limits me to what the distro offers, and makes modules with incompatible dependencies about as fun to deal with as a nest of wasps. But granted, it wasn't all bad, because at the very least, distro pkg managers automated dependency management and update things with a minimum of fuss (except when an update breaks something the package manager didn't install, and thus doesn't care about) in contrast to our next candidate, Manual installation.
Which also sucks, because depending on the source I was either dealing with a repo (which wasn't always git), a tar archive, and sometimes FTP servers. Some of these included C-libs. Some of those were precompiled. Some I had to build myself. Which ofc means I have yet another set of dependencies, but this time it's C. Integrating projects with with large dependency graphs? FUN (Short for "Frekked up nightmare") Sure, one could just vendor everything. That shifts the nightmare from build to update. No, wait, updating was ALWAYS a nightmare regardless, because without a package manager, there is no dependency management, so whenever anything was updated, it could potentially break something else. Manually checking the dependency graph for 42+ 3rd party modules, not all of which provided their dependencies in a standardized format? Not fun.
So yeah, by now I consider a package management system for a programming language essential. C/C++ excepted, because there the distros actually do a really good job of providing libs, because everything depends on them anyway.
My question was about MAINSTREAM PROGRAMMING LANGUAGES.
That is:
- Languages that actually matter as of right now, as opposed to being long irrelevant, extremely niche, hobbyist, esolang or not established.
- Languages that are not DSL
- Languages that actually have a concept of libraries and dependencies to begin with
So, with these clarifications, I repeat my question: May I have a list of 20 mainstream programming languages satisfying these requirements, that do not feature a package manager? Actually, tell you what, I'll lower the requirement to 15 languages, how's that?
To install without PIP: I download some python modules, put them in a directory/folder, and add that directory/folder name to the python sys path.
I do this when, because of old/small equipment, I am unable to use the encryption required by PIP, or total lack of networking.
When they work, PIP and Conda manage dependencies. Frequently they mis-manage dependencies: I can't believe how unbelievably bad they both are at managing dependencies: I can only believe that my use cases must be outliers. I guess that for people who aren't managing obscure modules on fleets of obscure systems, both must work ok.
Installing either way, either by hand or by using PIP, used to work much better before the python ecosystem acquired so many dependencies. PIP is a particularly obvious example, but lots of stuff I use requires TLS 1.3 and a handful of other library packages just to load, even for local filesystem operations, because it includes networking or authentication in the feature list.
Oh gd, so much this. Module A requires version X of module Z, but module B requires version Y of module Z. I need A and B, but version Y is not backwards compatible with version X and I can't have both. You can't resolve that with some godawful venv hack. You need a properly managed ecosystem.
I've done java maven and node npm and they are terrible and bloated.
a maven pom under 300 lines are seldom. and you need a artifactory or nexus local repo to build fast. and all the plugins.
npm a bit better but the directory stucture completely messy. typescrip ,
Maven has many faults but verbosity is not your enemy its your friend. There is nothing worse than a config file that has nothing and you or others think they know the defaults but it turns out the defaults are actually something else.
Defaults be it config or runtime are one of the biggest enemies of all software types, because we all know how fun it is when others make a decision on what they think is a good default and you end up with problems like the log4j exploits.
Writing everything down so you can read is always helpful.
I just watched the demo video and could not immediately see how it distinguishes itself from something like "pyenv".
Also, wasn't the premise the focus on packaging, which I would take to include distribution? I suppose that in the end I'm still building some wheel, pushing that out and good luck that you don't have dependency hell! Finally, at this early stage, there's no integration with anything from IDEs to various other frameworks that start with "something init" and dump metadata in your directory (*sigh* AWS CDK *sigh*)
Python is in a right mess, which is partly why I don't recommend it for actual software engineering.
Python is not helped by the fact that python gets used in Linux distros / desktops as a system component in a way that exposes that Python environment to users. Of course it's going to get mixed up with their dev python environments. It's amateurish design and it's asking for trouble.
But it's not that great elsewhere. If one looks at anything and everything from autotools to snap, there's some severe issues. Just try running two different versions of gcc on a system and build using the none standard one...
At least with OSes like MacOS, Windows and FreeBSD there's certain guaranteed minima to work to.
In Linux you can see why things like snap and flatpak get invented - about the only way of getting the right environment for software is to bring it with you these days - but feels like that is an insane way to go. Certain projects don't help either (cough GTK cough).
The entire Linux distro world should take a look at Visual Studio. If I install that, I don't end up with a build environment that's entangled with my OS and desktop environment in a tangled knot of conflicting dependencies. I don't end up with compiled code that is platform unique. I can very easily get a packaged installer that'll work on, well, any windows installation. And, if it comes to that, it also does a pretty good job of letting me compile, debug and run C/C++ software for Linux too (though it is then at the mercy of what is inside the Linux distro, but with WSL one can have quite a few different Linuxes at one's beck and call). It does this a lot better than most of the Linux dev tools (which is probably VS Code these days anyway).
With Linux distros as they are today, I really think that Linus and his colleagues in the kernel project are wasting their time. The kernel has done an admirably good job at keeping the kernel system interface consistent - "don't break userland". That's not achieving much if the all the "users" are going to do with software is either build it from source at installation, or run it inside a virtual machine. The kernel is written to support binary compatibility. The rest of the Linux world is so dysfunctional that that is an increasingly ignored and snubbed blessing.
Until Linux distros stop being structured to drive software building in a me, me, me and only me direction, software distribution will remain a messy nightmare. Build / dev environments need to be entirely separate to the OS environment.
Python is not helped by the fact that python gets used in Linux distros / desktops as a system component in a way that exposes that Python environment to users.
This is one of the many sins of the Linux distros. *BSD users understand the difference between system and user software and ports takes away all the pain that comes from conflating the two and lets you have pretty much all the versions of a language that you would like. On my Mac I have Python 3.6 to 3.12 and no conflicts.
> Of course it's going to get mixed up with their dev python environments.
That is a long-solved problem, even ignoring virtualization entirely. `pyenv` provides shims and multi version installs that don't interfer with the OS's python installation. `venv` provides environments ie. package-isolation within these installations. Both can be used in concert to provide as many arbitrary combinations of interpreter and modules as I want, all on the same machine, without any virtualization, without any interference with the systems Python install.
And again, that is before we even mention containers, which achieves the exact same effect, only in a vastly more convenient manner, that applies to every language and build system, not just python.
If people run python projects in their production environment without these, that's their own problem. If I insist on throwing 40 kg of camping gear in my trunk and then find my fishing lines tangled with my socks and never see half the tent-pegs again, I have only myself to blame for not packaging properly.
> Just try running two different versions of gcc on a system and build using the none standard one...
https://hub.docker.com/_/gcc
You were saying?
> but with WSL one can have quite a few different Linuxes at one's beck and call
docker run -it debian /bin/bash
docker run -it archlinux /bin/bash
docker run -it redhat/ubi8 /bin/bash
docker run -it alpine /bin/ash
You were saying?
> The rest of the Linux world is so dysfunctional that that is an increasingly ignored and snubbed blessing.
Curious then, isn't it, that this "snubbed blessing" is powering almost the entirety of this planets sever infrastructure, as well as the vast majority of mobile personal tech from smartphones to tablets.
> Build / dev environments need to be entirely separate to the OS environment.
No they don't, because it's my environment, and I want to be able to make it as tidy or chaotic as I like. And because I can do all that, I can, on my machine, using one-liner terminal commands, spin up not one, not two, but an arbitrary amount of different dev/build/test/production environments, in seconds, up to and including simulating entire networks of microservices.
> "A class of users expect a packaging tool that provides a cohesive experience (like npm (NodeJS), cargo (Rust), gem (Ruby), pub (Dart), dotnet (C#/.NET), etc)
Yes, and what do most of such tools have in common? They were there for almost the whole lifetime of the project. That's why almost every tutorial, book, video, etc. on the topic features them.
Sorry if I sound like a negative nelly here, but that ship hasn't just sailed for Python, it has sailed, came back, shanghaied a new crew, was sunk with a load of sugarcane, tobacco and spice, was risen from the depths by an ancient carribean curse, and is currently headed for Tortuga.
This is something that projects have to get right from the beginning. If they don't then it doesn't matter how many new solutions there will be, or how good they are, the genies won't go back into their bottles. Look at C. How many different systems for building, dependency management, etc. are there for C? How unified is the landscape? And C had a lot more time to come up with good solutions for this. And it did. But it didn't help, because they were not there from the beginning, and by the time good tools came into existence, they were just more programs competing in the same mindspace.
In python the situation is very similar. Like C, Python became an integral part of the Unix Landscape. Every distro packages it, so there are not just competing build systems, there is also the systems own package manager with its strong opinion about how things should be.
None of that means that this project isn't a good idea. I absolutely like Ronachers work, and will definitely give Rye a whirl. But I don't think that this will solve the headaches of python deployement. It will only make it somewhat less of a headache.
Yeah, that one I don't get either.
I mean, sure, a lot of people will just load the compiled binaries. However, some people will want, or need to, build it from scratch, and pulling in the entire rust buildchain, may not be a problem when the system is already using rust...but if it doesn't, and that's not an unlikely scenario, then requiring its build chain and libraries to bootstrap something in python, when every other management tool for it is built either in python itself, or lives in a C-library called from python code ... I mean, why?
...may have solved all of these issues in the most elegant way out of all mainstream languages?
Because, what's a golang module? A repository with some special plaintext files in it. aka. where the code will eventually end up anyway. Packaging a golang module for others to use, happens the moment I press ENTER after typing `git push`.
What's the import-path? Why it's a simple string that contains all the information (aka. the URL and package name) required for the toolchain, which lives in a single executable that ships with the language core, to find and download modules using tools that already exist...like git and mercurial.
Dependency management? Transient dependencies are directly defined in the source. When I want to import package X, and X wants to import Y, simply reading the source tells the toolchain all it needs to know about transient dependencies, simply because of how import-paths work.
Version & integrity control? The aforementioned plaintext files contain versions (go.mod) and checksums (go.sum). The toolchain handles both of them for the developer. And how are versions defined? Why, again using a system that already exists: semantic versioning via git-tags.
Sorry for sounding like a massive fanboy, but this. is. just. brilliant. The entire ecosystem works without requiring a central package authority. Builds work using just the info in the source files + go.mod/sum. Yes, goproxys exist, but they are not needed for the system to work. As long as peoples repos are up, the ecosystem is up. And the toolchain can make use of a lot of capabilities and semantics that already exist independent of golang, put them together in a clever way, and throw the end-result into the compiler.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
