undefined | Better HN

0 pointsmicrotonal2y ago0 comments

Upstream Hydra doesn't build packages with CUDA because it uses a non-FLOSS license. So they are not in the binary cache. You'll end up rebuilding every CUDA-using package every time a transitive dependency is changed. Yeah, I know, pin the world. But you'll still have to build these packages on every machine. So, you have to run your own binary cache. As you see, the rabbit hole gets deep pretty quickly.

The only recourse is using the -bin flavors of PyTorch, etc. which will just download the precompiled upstream versions. Sadly, the result will still be much slower than other distributions. First because Python isn't compiled with optimizations and LTO in nixpkgs by default, because it is not reproducible. So, you override the Python derivation to enable optimizations and LTO. Python builds fine, but to get the machine learning ecosystem on you machine, Nix needs to build a gazillion Python packages, since the derivation hash of Python changed. Turns out that many derivations don't actually build. They build with the little amount of parallelism available on Hydra builders, but many Python packages will fail to build because of concurrency issues in tests that do manifest on your nice 16 core machine.

So, you spend hours fixing derivations so that they build on many core machines and upstream all the diffs. Or YOLO and you disable unit tests altogether. A few hours/days later (depending on your knowledge of Nix), you finally have a built of all packages that you want, you launch whatever you are doing on your CUDA-capable GPU. Turns out that it is 30-50% slower. Finding out why is another multi-day expedition in profiling and tinkering.

In the end pyenv (or a Docker container) on a boring distribution doesn't look so bad.

(Disclaimer: I initially added the PyTorch/libtorch bin packages to nixpkgs and was co-maintainer of the PyTorch derivation for a while.)

0 comments

alchemist1e92y ago

As a heavy nixpkgs user your comment resonates and makes me nervous.

I was thinking if it is possible in nixpkgs to create a branch that attempts to create a version match to specific distributions, especially Ubuntu as the ML world is most using it. My idea is to somehow use the deb package information to “shadow” another distribution.

> First because Python isn't compiled with optimizations and LTO in nixpkgs by default, because it is not reproducible. So, you override the Python derivation to enable optimizations and LTO. Python builds fine, but to get the machine learning ecosystem on you machine, Nix needs to build a gazillion Python packages, since the derivation hash of Python changed. Turns out that many derivations don't actually build. They build with the little amount of parallelism available on Hydra builders, but many Python packages will fail to build because of concurrency issues in tests that do manifest on your nice 16 core machine.

I understand your comments including above and the one about CUDA binaries. Just one clarification on the concurrency in tests failure, do you mean it overloads the machine running multi process tests that then tests fail due to assumptions by the package authors?

My main point is that Nix as a system is so incredibly powerful that perhaps there is an ability to “shadow” boring distributions, especially debian based, in some automated way. The we would have the best of both, baseline stability from the distribution and extensibility of nix.

microtonalOP2y ago

I've found that quite some test suites have race conditions (e.g. simultaneous modification of files, etc.), which manifest themselves e.g. when a package uses pytest-xdist (and the machine has enough cores).

My main point is that Nix as a system is so incredibly powerful that perhaps there is an ability to “shadow” boring distributions,

I think things would improve vastly if it was possible to do CUDA builds in Hydra and have the resulting packages in the binary cache. My idea (when I was still contributing to nixpkgs) was to somehow mark CUDA derivations specially, so that they get built but not stored in the binary cache. That would allow packages with CUDA dependencies to get built as well (e.g. PyTorch). Nix would then only have to build CUDA locally (which is cheap, since it only entails unpacking the binary distribution and putting things in the right output paths) and would get everything else through the binary cache (like prebuilt PyTorch). But AFAIR it'd require some coordinated changes between Nix, Hydra, etc.

Then I started working for a company in the Python/Cython ecosystem and quickly found out that Nix is not really viable for most Python development. So I am now just using pyenv and pip, which works fine most of the time (we have some people in our team who are very good at maintaining proper package version bounds).

alchemist1e92y ago

Have you ever come across anything debian -> nix expression tools?

Ubuntu seems to be winning mindshare across the board and while this would be different than nixpkgs itself I was thinking if it is possible to mass convert deb packages into nix expressions, this combined with overlays would allow rapid incremental testing of marginal modifications to a current distribution’s stacking of versions.

A bit like how Nix community has tools on top of the various language packaging systems but this would be a layer on top of the debian packaging standards.

Maybe it’s crazy but just an idea I’ve been having recently and wondering how hard it might be. Importantly debian deb and apt systems are very reproducible and structured which is a good fit for a Nix based layer.

rch2y ago

Granted, but I already have to run isolated container registries, pypi, maven, terraform, CI/CD, etc., etc., and so locally addressing the problems you've described is unavoidable and will realize significant efficiencies in any case. Everything about working in partially or fully air gapped environments is painful - no surprises there.

But I also think it's fine for individuals and researchers working in ML to expect some extra compiling, as long as the outcome is reliable. I'm stuck at home this weekend resurrecting an analysis from 10+ years ago, complete with Python, R, Java, and Fortran dependencies^, and I'm definitely wishing I'd known about Nix back then.

^btw, thanks to whomever included hdf5-mpi in Nixpkgs. Your work is greatly appreciated.

rch2y ago

I'll add that I do end up disabling tests more than I'd like, even for very popular packages. Point taken there.

UncleEntity2y ago

I learned the very hard way not to mess with the python version the system depends on.

If you absolutely must then build it separately and link (or use) that exactly like blender does with their binaries. Campbell (one of the core blender devs) used to love to bump the python version as soon as it was released and if you wanted to do any dev work you’d have to run another python environment until the distro version caught up. Being as I liked to use the fedora libs as a sort of sanity check this was a bit of a hassle to say the least.

microtonalOP2y ago

I learned the very hard way not to mess with the python version the system depends on.

That's not much of an issue on Nix. You can just override Python for your particular (machine learning) package set. The rest of the system will continue to use Python unmodified.

j / k navigate · click thread line to collapse

0 pointsmicrotonal2y ago0 comments

In the end pyenv (or a Docker container) on a boring distribution doesn't look so bad.

(Disclaimer: I initially added the PyTorch/libtorch bin packages to nixpkgs and was co-maintainer of the PyTorch derivation for a while.)

0 comments

alchemist1e92y ago

As a heavy nixpkgs user your comment resonates and makes me nervous.

microtonalOP2y ago

My main point is that Nix as a system is so incredibly powerful that perhaps there is an ability to “shadow” boring distributions,

alchemist1e92y ago

Have you ever come across anything debian -> nix expression tools?

A bit like how Nix community has tools on top of the various language packaging systems but this would be a layer on top of the debian packaging standards.

rch2y ago

^btw, thanks to whomever included hdf5-mpi in Nixpkgs. Your work is greatly appreciated.

rch2y ago

I'll add that I do end up disabling tests more than I'd like, even for very popular packages. Point taken there.

UncleEntity2y ago

I learned the very hard way not to mess with the python version the system depends on.

microtonalOP2y ago

I learned the very hard way not to mess with the python version the system depends on.

That's not much of an issue on Nix. You can just override Python for your particular (machine learning) package set. The rest of the system will continue to use Python unmodified.

j / k navigate · click thread line to collapse