I don't say this out of negativity for the sake of negativity. Earlier today, I was trying to resurrect an old Python project that was using pipenv.
"pipenv install" gave me an error about accepting 1 argument, but 3 were provided.
Then I switched to Poetry. Poetry kept on detecting my Python2 installation, but not my Python3 installation. It seemed like I had to use pyenv, which I didn't want to use, since that's another tool to use and setup on different machines.
I gave up and started rewriting the project (web scraper) in Node.js with Puppeteer.
What people casually think of as "Python" is really a huge dynamic ecosystem of packages. Imagine that there are 60k packages, each with 1k lines of code... that's a 60 million line codebase, and it can't be checked for breaking changes. Short of continually testing your code against the latest versions of packages, you're going to hit some bumps if you haul an old code out of the vault and try to fire it up on a new system.
I don't know how Javascript developers handle this.
I handle it by running Python inside an isolated environment -- WinPython does this for me -- and occasionally having to fix something if a new version of a package causes a breaking change.
The drawback of my method is deployment -- there is no small nugget of code that I can confidently share with someone. They have to install their own environment for running my stuff, or take their chances, which usually ends badly.
Most package managers have lockfiles that allow for some degree of determinism. Of course if a library introduces breaking changes you're going to have to rewrite, but only when explicitly upgrading the dependency.
That said, I 100% agree. I don't have the answer, except that I wish that there was one official answer, for developers and deployment that was easy to explain to beginners.
For what it's worth I've been using pipenv for over a year and it works good enough. I think npm's approach is better, but not perfect. I've heard good things about yarn. I know CPAN is the grandfather of all of them. I've barely used gems but they seem like magic, and Go get your repo URI out of my code please and thank you. :-) All fun aside, what languages have it right? and is there maybe a way to come up with a unified language dependency manger?
Technically, OP didn’t have to use pipenv for anything except knowing which versions of each dependency to install (Pipfile.lock) with good old pip. Those other tools are mere conveniences. Giving up on a language for that...that’s drastic.
For interpreter/compiler version management you can use asdf [0]. It works for all popular programming languages. You can use it to replace tools such as pyenv, nvm, gvm, etc.
Everything is pinned to exact source revisions. You can be relatively sure to be able to git clone and nix-shell and be off to the races.
You can even go the extra mile and provide working editor integration in your nix-shell (especially easy with emacs). So you can enable anyone to git clone, nix-shell, and open a working editor.
The biggest downside is becoming proficient in Nix isn't easy or even straightforward.
While you're here, how do you do this with nix-pkgs?
I looked into using nixos for nodejs deployments recently, and was amazed to find that the versions of node in the nix-pkgs repo are just pinned to X.Y.0 releases, with no discernable way to update to a bugfix release after .0 , so... I don't see how this could possibly be used for production deployments?
https://nixos.org/nixos/packages.html?channel=nixpkgs-unstab...
What I really want is to be able to tell nix to give me nodejs at version 10.16.3 and have it do the right thing every time.
I'm happy to be wrong on any of this.
Occasionally I still run into weird non-deterministic issues with builds, though: https://github.com/NixOS/nixpkgs/issues/71178
You'd think "pinning" a python version + channel would avoid this.
In Python it's a mess. Some packages specify their deps in setup.py; some in a requirements file, which may or may not be read in by their setup.py. It's not rare to have to have multiple 'pip install' commands to get a working environment, especially when installing fairly boutique extensions to frameworks like Django.
There just isn't a holistic, opinionated approach to specifying dependencies, especially at a project level. Which leaves unexpected upgrades of dependencies (occasionally leading to regressions) as a reality for Python devs.
One is pipenv [0], which works similar to yarn & co. It uses Pipfile/Pipfile.lock to define and lock dependencies. Pipenv has a major flaw: It can't be used to publish packages on pypi.org (you still need twine & setup.py for that). It's also known for being slow and somewhat buggy. Despite all that pipenv is an "official" tool maintained by the "Python Packaging Authority".
The other one is poetry [1], which works exactly like yarn & co. It uses "pyproject.toml" to specify dependencies and "poetry.lock" to lock them. Poetry does most of the things "right", but it's still an underdog compared to pipenv.
Both tools have not yet fully matured, thus there are a lot of complaints.
Now try that with JavaScript or Rust. If you don't fix versions, come back three months later and compatibility is usually fucked up beyond all recognition.
Some languages embraced better dependency locking because they absolutely couldn't not solve the problem.
pipenv generates a Pipfile.lock - if it can, I've used it primarily for Airflow, and some packages within Airflow have incompatible version ranges for the same dependency, which means it can't generate the lock file.
npm sucks and all, but at least it just works and doesn't get in my way as much.
used many package managers: pip, gem, go, npm, composer, etc... npm is the only one i have recent memories of having to kill & the only one that makes the fan go off (well okay most c/c++ do that too...)
quite frankly surprised by what i am seeing about python here. i have never been into the new stuff, pipenv, poetry, pipx, etc... maybe that's where the bad experience is coming from? i even don't know when and why it got so complex...
Like any dictatorship, that can be fine if those in charge are benevolent and competent. For programming languages, the first is almost always true, but the second is far from guaranteed. Skill at programming language development has no bearing on skill at developing a build, packaging, and distribution system.
Go is a prime example of this. The language is pleasant to use with few ugly surprises, but their build system has been awful for a decade, only now reaching a semi-decent state with modules (which are still pretty damn ugly).
With python, on the other hand, there's competition in this space, and as a result the tools are pretty nice, albeit fragmented.
But then there's rust, which has a nice language AND a nice build system. You take a big risk when building both the language and build system; sometimes it works, sometimes it doesn't. And you risk fragmentation if you don't. It's a tough choice.
python3 -m venv /tmp/foo
/tmp/foo/bin/pip -U pip wheel
/tmp/foo/bin/pip -r requirements.txt
I understand some might not like it, but really, it's simple and it works.
Tkinter and mpi4py are the most recent ones I've had this problem with. I expect someone will tell me "it's trivial to install these in a venv, just do X", but X is not obvious to me.
Having said that: you generally want this packages integrated with your system (which provides a self-consistent Tcl as well as MPI-environment)
It’s super useful for maintaining static versions of python, like 2.7, 3.6 and 3.7 when you have many projects that have different python requirements.
If you're working on closed-source code or have tight control of your environment, it's enough to develop and run on a single version, rendering pyenv and whatnot unnecessary.
This post reads most to me as fad. It is an opinion, you would most likely bail out.
Very few issues arise from our choice of build tooling. Not enough to consider switching at the moment. I suspect I'll try pyenv next time I have a new dev maching to set up but only as it seems fairly painless to switch from virtualenv.
It just shows that Python is used for a lot of purposes and there is no single tool that handles all usecases.
> I gave up and started rewriting the project (web scraper) in Node.js with Puppeteer.
And when you'll get to the point where you need to work with different node projects, you will also need several tools to manage node versions and different environments, so that doesn't help at all.
* "I need to develop against multiple Python versions - various Python 3 versions (3.6, 3.7, 3.8, mostly), PyPy, and occasionally Python 2.7 (less and less often, thankfully)."
* "I work on many projects simultaneously, each with different sets of dependencies, so some sort of virtual environment or isolation is critical."
So sure, there is more than one way to manage a Ruby runtime version, but are there any competitors to Ruby's Bundler? I feel like it's the undisputed champion of Ruby dependency management, working with Rubygems, another unopposed incumbent in its own space, and it never even occurred to me that my language should have more than one of either such tool. Can someone help me understand what drove the Python ecosystem to need more than one dependency manager, ELI5?
I am pretty cloistered as a Ruby developer, but my limited professional experience with other language runtimes tells me that the Ruby "developer environment" experience is really second to none, (change my mind.) Is there any tool which is nearly comparable to Ruby's Pry for Python or other interpreted languages? (I doubt that there is!)
Managing gemsets and multiple versions of Ruby is old-hat.
poetry == npm
pyenv == nvm
pipx == npx
No big difference, IMO.
Package Management - npm? bower? yarn? Which should I use this week?
Interpreter Versions - Revisiting a project I last touched 2 years ago on node 8 has a load of locked dependencies that only work on that node version. OK, let's bring in nvm so I can get a dev environment going.
Executable Packages - oh no I've got two different projects with different and incompatible versions of apollo, what do I do? Oh right, sure npx for executable isolation so we don't pollute the global namespace.
Every ecosystem has these problems, and if they don't it's probably because they're still relatively esoteric.
Since Python did not prescribe a way to handle it the community has invented multiple competing ways to solve the problem, most of which have shortcomings in one way or another.
To further add to the confusion, most Linux and Unix-based operating systems (Linux, MacOS, etc.) have their own system Python which can easily get fouled up if one is not careful.
This is one place where Java's use of a virtual machine REALLY shines. You can build an uberjar of your application and throw it at a JVM and (barring JNI or non-standard database drivers) it just works. There is also usually no "system Java", so there is nothing to break along those lines.
Exactly. It's not Python's only problem, but far and away the most painful snags I've hit with packages is when they use C code, and thereby drag in the whole system. "I'll just `pip install` this- Oh, I need to install foo-dev? Okay, `apt-install foo-dev`... oh, that's in Ubuntu but not Debian? Well this is gonna be fun..." Now I trend a bit more exotic in my systems (NixOS, Termux, *BSD, ...) but if my Python were just Python it would just work so long as I have a working Python install; in practice that's rarely enough.
Oh, but there is :( `jenv shell 1.8` is muscle memory for me now.
Given that: are you sure problems arise around pure python-packages (which generally have a well enough forward compatibility), or is the problem with all the cool "machine-code"-embedded packages (which are a lot!)? And yeah, these might indeed break randomly, when installed on different systems of different time-periods. But that's a problem all binary packages have!
There is a lot of tools and confusion between versions especially because of the breaking changes between v2 and v3 (try to explain someone why he has both and when he types python it will take v2.7 as default).
I love the elegance and simplicity of the language and many tools written with it but this is a point I'd really much appreciate to be improved.
Because of that I sometimes just rewrite something before fixing it with 2.7. It's perhaps a bit more work sometimes but not as frustrating as trying to get something running that is deprecated in parts.
In Smalltalk you change parts of the environment. In Python, Java (and Ruby?), you change the entire environment, as described in TFA.
./gradlew run
Haskell with new-style cabal works like a charm with `cabal init` for package setup. And then ghci from there will give you...
- expression evaluation ofc
- type-at-point via type hole annotations
- Laziness inspector via :print
- Breakpoint debugger (someone just posted a nice Reddit text post about it today)
- Package information of terms via :info (and :load $MODULE to get any top-level term in $MODULE into repl scope)
- Docs via :doc (this is pretty new)
- Module export lists via :browse
off the top of my head
$ curl -sSL https://get.haskellstack.org/ | sh
$ stack install turtle
$ cat >> ./hello.hs <<EOL
#!/usr/bin/env stack
-- stack --resolver lts-14.14 script
{-# LANGUAGE OverloadedStrings #-}
import Turtle
main = echo "Hello, world!"
EOL
$ chmod u+x ./hello.hs
$ ./hello.hs
Hello, world!
On the contrary, I seldom had these kind of issues with projects coded in C#, C or C++: most of the time the few steps to compile the project succeeded and produced a usable binary.
As far as virtual enviroments go, I actually kind of like them. They were containers before containers became a thing, and I’ve had much fewer issues with them than say NPM, but they are more burdensome than adding full binaries to your C# project.
Where compiled languages shine, and maybe I’m misunderstanding you, is when you need to distribute and executional to end-usere. C# is much better at that than Python, but we haven’t actually done that at my shop, in around 10 years.
That said, once you get it down, you're not burdened by it much/at all. You can start and begin work on a new project in seconds, which feels important for a language that prides itself on ease of use.
But yeah, not a great look for newcomers.
Nowadays, using setuptools to create packages is really easy too, there's a great tutorial on the main Python site. It's not as easy as node.js, sure, but there's tools like Cookiecutter to remove the boilerplate from new packages.
requirements.txt files aren't very elegant, but they work well enough.
And with docker, all this is is even easier. The python + docker story is really nice.
Honestly I just love these basic tools and how they let me do my job without worrying about are they the latest and greatest. My python setup has been stable for years and I am so productive with it.
From there it's just "workon projectname" and just "deactivate" when I'm done (or "workon otherprojectname")
It has been stable and working for ages now. I just don't see any strong incentive to change.
Isn't this just a variant of what the original comment is critiquing, though? In order to sanely use Python and external dependencies you need some highly stateful, magical tool that does 'environments' for you. The conceptual load of this is quite high. Adding docker only makes it higher - now you're not merely 'virtualizing' a single language runtime but an entire OS as well - just to deal with the fact your language runtime has trouble handling dependencies.
(and for all npm-friends: it's exactly the same "conceptual load" which arises from having node 8 and 12 around. except that these will be just incompatible by default, so noone bothers)
You can use `pip-tools` to get something like a Gemfile/package.json, but there are a few restrictions that are suboptimal.
So Pipenv/Poetry are the current best ways to get something like the package management story that other languages like Ruby and JS have had for a long time (and that Go's been polishing recently too).
Being from JAVA Shop for long time, If I have to switch between different version of JAVA, all I do is change JAVA_HOME to point to correct version, go to base project directory and "mvn clean install" does the job. :).
The one thing I thought was neat here was pipx. I do have a few CLIs set up in my default conda env and haven't run into any dependency problems yet, but have occasionally tried to use them while another env is activated. Having a separate env automatically created for the entry points is a nice value add.
Need to have multiple versions of python installed and easily accessible? Use pyenv.
Need to run tests across multiple versions of python? Use tox.
Need to freeze environments for deployment purposes? Use pip-tools.
Need to freeze the entire operating system? Use docker or vagrant.
Don't use tools you don't need. That would be silly.
However, you can run into issues if you are using different versions of Python, or Python on different operating systems.
I mean compared to Java and C# I have a MUCH MORE EASIER time to set up my development environment. Installing Python, if I am on a Windows box I mean, is enough to satisfy a lot of the requirements. I then clone the repo of the project and
source venv/bin/activate
pip install -r requirements.txt
is enough to get me to start coding.
Because it's more complex if you have projects on multiple Python versions and if you want to lock your Python packages to specific versions. (Pip can bite you when different packages have different requirements for the same lib).
https://github.com/sdispater/poetry/issues/571
the OP himself has a fix for this in his own dotfiles repo:
https://github.com/jacobian/dotfiles/commit/e7889c5954daacfe...
(Though, how'd you find that? Mildly creepy that you know more about my dotfiles than I do!)
How so? I've been using Docker for development for years now and haven't experienced this EXCEPT with some slowness I experienced with Docker Compose upon upgrading to MacOS Catalina (which turned out to be bug with PyInstaller, not Docker or Docker Compose). This is on a Mac, btw; I hear that Docker on Linux is blazing fast.
I personally would absolutely leverage Docker for the setup being described here: multiple versions with lots of environmental differences between each other. That's what Docker was made for!
My experience has been that it's significantly more effort to meet my requirements with Docker, and that I spend a _lot_ of time waiting on Docker builds, or trying to debug finicky issues in my Dockerfile/docker-compose setup.
I'm sure all of these things have fixes -- of course they do! But I find the toolset challenging, and the documentation difficult to follow. I'd love to learn what I'm missing, but I also need to balance that against Getting Shit Done.
What's the situation where your application needs to go first in the Dockerfile, and then you need to put a bunch of stuff that doesn't depend on your application?
I find it useful to keep a WinPython installation on a flash drive in my pocket. I can plug it into somebody's computer and run my own stuff, without worrying that I'm going to bollix up their system.
I did a quick comparison here[0], and I'm planning to do an update with the latest version of Poetry.
[0] https://johnfraney.ca/posts/2019/03/06/pipenv-poetry-benchma...
But what really pushed me away is that installing or upgrading any single package upgraded all dependencies, with no way to disable this behavior. (I believe you can now.) A package manager should help me manage change (and thereby risk), not prevent me from doing so.
Poetry is probably the best of the all-in-one solutions. It does its job well but I've found the documentation lacking.
In the end, I've settled on pyenv-virtualenv to manage my environments and pip-tools to manage dependencies. It's simple and meets my needs.
Now that poetry also manages virtual environments it’s far and away the better choice.
Caveat - Heroku doesn’t understand pyproject files yet, so no native poetry integration. Heroku are working on this.
The main reasoning was so that I could easily build and publish packages to a private repository and then easily import packages from both pypi and the private repository.
Happy to answer more questions.
We have a custom pypi server and need all requests to go through it, however haven't figured a way to make poetry always use our index server for all modules instead of pypi.org
I've flagged trying to manage python versions on my machine and just develop inside docker containers now (Thanks to VSCode). Using tightly versioned python base images
That said, all I want is for a unified standard to emerge, this is getting a little ridiculous...
I never manged to hose the OS Python on Linux, by sticking to a very simple rule: DON'T BE ROOT. Don't work as root, don't run `sudo`.
On Linux, I use the system python + virtualenv. Good enough.
When I need a different python version, I use docker (or podman, which is an awesome docker replacement in context of development environments) + virtualenv in the container. (Why virtualenv in the container? Because I use them outside the container as well, and IMHO it can't hurt to be consistent).
python -m venv venv
source venv/bin/activate
pip install -U pip
pip install whatever
# <do you stuff here>
deactivate
[1] https://docs.python.org/3/whatsnew/3.4.html#whatsnew-pep-453
And, given that I've been trying NixOS lately and had loads of trouble and failing to get Conda to work, I will definitely give this setup a try.
(I haven't quite embraced the nix-shell everything solution. It still has trouble with some things. My current workaround is a Dockerfile and a requirements.txt file, which does work...)
1. Highlight to run 2. Remoting into a kernel
Both features are somewhat related. I want to be able to fire up a Python Kernel on a remote server. I want to be able to connect to it easily (not having to ssh tunnel over 6 different ports). I want connect my IDE to it and easily send commands and view data objects remotely. Spyder does all this but its not great. You have to run a custom kernel to be able to view variables locally.
Finally, I want to be able to connect to a Nameko or Flask instance as I would any remote kernel and hot-swap code out as needed.
I gotta give poetry a try, though.
In principle, it's a good idea; in practice, I'm not satisfied. On Windows, it's an easy solution, especially for packages that depend on non-python dependencies (e.g. hdf5).
Can anyone comment on the Docker learning & troubleshooting story for python?
You'll get confident and from there learning `docker-compose` is a breeze.
It feels pretty comfy to effectively be on an island and far away from the hustle and bustle of the industrial Python tooling.
1. Install Anaconda to your home user directory .
2. create environment using (conda create --name myenv python=3.6) .
3. Switch to the environment using (conda activate myenv) .
4. Use (conda install mypackage), (pip install mypackage) in that priority order .
5. Export environment using (conda env export > conda_env.yaml) .
6. Environment can be created on an other system using (conda env create -f conda_env.yaml) .
Anaconda: https://www.anaconda.com/distribution/#download-section .
Dockerized Anaconda: https://docs.anaconda.com/anaconda/user-guide/tasks/docker/ .
One pain point though: getting it to work with Sublime Text 3 requires you to set the `CONDA_DLL_SEARCH_MODIFICATION_ENABLE` environment variable to `1` on Windows.
Not a flaw of Anaconda: it just pays attention to how to with multiple Python installations on Windows.
This difference is important due to dependencies- if you have two different CLI tools you want to install but they have conflicting dependencies then pip is going to put at least one of them into an unusable state, while pipx will allow them to both coexist on the same system.
pipx, poetry, pipenv and co are still nice wrappers to have, I suppose. It just feel less useful now that most of my projects are dockerized.
Doesn't seem fair. You're not abandoning requests, are you?
Edit: From https://www.python.org/psf/github/,
> ... we have created a GitHub organization, @psf, to support and protect projects that have outgrown ownership by their original author.
2. Conda is not used as much as you might think... it's really only used within the data science community.
2. Conda is quite widely used outside of data science. It’s for example part of Anaconda enterprise offerings used by huge banks, government agencies, universities, etc., on large projects often with no use cases related to data science. Conda itself has no logical connection with data science, it’s just a package & environment manager.
In each of my last 4 jobs, 2 at large Fortune 500 ecommerce companies, conda has been the environment manager used for all internal Python development. Still use pip a lot within conda envs, but conda is the one broader constant.