undefined | Better HN

0 pointsapi2y ago0 comments

As someone who does this for a lot of distros (ZeroTier), I can say that it is hell and I understand why devs don't want to do it.

Two specific examples:

(1) For some reason a lot of Debian distros decided to rename and/or re-version-number OpenSSL for no good reason (pedantry is not a good reason), meaning if you depend on OpenSSL/libcrypto your packages will mysteriously break all over the place because you're not using the versioning scheme or naming convention. We're not doing it now but we may switch to statically linking this.

(2) We use a UPnP library called miniupnp in the current version. We have to statically link it because of issues like (1) and also because some distros were for a time stuck on a version that had security bugs that they would not upgrade because it would break other packages.

So imagine (1) and (2) ad infinitum times a hundred little distributions and Debian forks and... it's completely untenable. Static linking solves many of the problems but that defeats some of the purpose of package management.

We do it for all the major Debian and Ubuntu distributions by using a qemu-chroot based build farm that builds on each distribution+architecture combo with actual qemu binary emulation. It's over 200 gigabytes of chroots and takes two hours on a 64-core Threadripper to build all the artifacts. We tried cross compilation and had too many problems: it builds fine, seems to run fine, then users complain that some mysterious glibc symbol was missing on their system. We switched to building on actual chroots and most of those problems went away. Most of them.

Snap, FlatPak, and Docker are all variations on the same basic conclusion that many developers reached long ago: "fuck it, just distribute software in the form of tarballs of entire Linux installs."

The only thing worse is Windows MSI installers and Windows drivers, though with those at least there's less platform variation so once you battle those horrors the result mostly works on most systems. Mostly. But whoo boy is the Windows installer system horrific. I shall quote Pinhead: "I have such sights to show you..."

Apple is the easiest because they have less legacy baggage than Windows and only one "distribution." They do have some horrors around signing and notarization and we don't distribute in the Mac App Store (yet?).

A major reason SaaS delivered through browsers is eating the world is the immense and totally unnecessary pain of distributing apps for major OSes. Even Apple is quite a bit harder than making a web site that you visit in a browser. All this pain is 100% the fault of OSes not caring about developer experience and in the case of Linux the multiplication of endless "vanity distributions" to no benefit to the community.

If you want packages instead of tarballs of OSes, Linux distributions could (1) consolidate and (2) invest some time in the very boring totally un-sexy work of massive amounts of clean-up and standardizing things as much as possible. But it's more fun to make another vanity distribution I guess. You'll get it right this time and everyone will switch to your distribution.

0 comments

Izkata2y ago

> Snap, FlatPak, and Docker are all variations on the same basic conclusion that many developers reached long ago: "fuck it, just distribute software in the form of tarballs of entire Linux installs."

..wait a minute

> (2) We use a UPnP library called miniupnp in the current version. We have to statically link it because of issues like (1) and also because some distros were for a time stuck on a version that had security bugs that they would not upgrade because it would break other packages.

Wouldn't this solve the same problem without adding snap/etc?

apiOP2y ago

It solves some of the problems, but not all, and it requires a lot of testing and trial and error and setup. Sometimes libraries are hard to statically link. Sometimes you have problems with different glibc versions.

I forgot to say: some distros that people still use are too old for us to build for, so we build fully static binaries for those. But you can't really statically link glibc, so we build them on Alpine statically linked with musl. That usually works okay. It's the best we can do for the old stuff.

We've invested a lot of time in this. For most developers I totally understand "fuck it."

account422y ago

I have had a lot of success with building against old glibc version and then bundling all required libraries except system ones (glibc, libGL, libX11 etc). Anything optional needs to be loaded with dlopen/dlsym but for many things that detail is already handled by libraries like SDL. That's generally what seems to be the common solution for games (well used to be at least, today most just build against the Steam runtime and call it a day). It's really not that different from Windows IMO except that the set of base system libs that you can rely on to be backwards compatible and don't need to bundle is smaller.

You can still pacakge that distro-agnostic build up into distro-specific package formats to make updates easier for users without needing zillion of different build roots just for Linux.

j / k navigate · click thread line to collapse