https://search.nixos.org/options?channel=21.11&show=boot.bin...
https://github.com/NixOS/nixpkgs/blob/111839dcf6e9a8bac6972e...
The next step is to do the same thing except using containers/namespaces. I was able to run a Yocto rootfs build for ARM completely, including init, and IIRC networking, using LXC and binfmt_misc. A very handy technique for testing and it does run much faster than full-system emulation.
Now I found a way to get arm64 crystal binaries. I got rid of the container, but Invidious still cross compiles to amd64, so qemu is still used to run an amd64 build Invidious of transparently.
binfmt and qemu-user do wonders. It works well. One could use box64 [1] instead of qemu and it should provide better performance because it uses the native versions of some well known libraries (including libc6) instead of emulating them, but I failed to compile box64 this weekend so I stayed with qemu.
I remember http://scratchbox.org/ would allow you to replace some components (e.g. gcc) with their native versions so as to speedup them. It is all hopelessly broken now.
With a little work you should be able to do this as unprivileged/rootless as well.
I'd started out using a full emulation VM, and it was alright, but the cost of emulation was crippling for parts of the build process. IIRC one part of the build process was pulling in python libraries that didn't have arm wheels, and that took a bit of work to compile even on native architecture. Add in the overhead of full system emulation and it really hurt the iteration process. Especially as I worked my way from "Finally got it to build!" through to "Got the build repeatable from scratch!"
The binfmt / container approach dramatically reduced the amount of emulation being done, resulting in phenomenally faster build times.
Then I finally got access to an actual Arm instance and the entire process took even a fraction of that time.
The opposite is true if you're virtualizing the same architecture as the host with hardware virtualization (KVM) enabled. Counter-intuitively, user emulation is much slower than full-system emulation in this specific case.
I'd love to try that! Any pointers? :)
- build a rootfs using the poky reference distro (but do it for your arm target).
- https://docs.yoctoproject.org/
- https://docs.yoctoproject.org/brief-yoctoprojectqs/index.htm...
- you'll need to make or get a layer for your machine type. for example, for rpi you'll want: https://github.com/agherzan/meta-raspberrypi
bitbake core-image-minimal (or whatever the machine layer wants you to do)
- run (note: in the past I used LXC, but lets try podman today)
apt install qemu-user-static podman
podman run -it --rootfs <rootfsloc> /sbin/initSecond: of you like playing with foreign architectures, I have a collection of ready to boot Debian images of many architectures, that you can promptly boot with QEMU. Command line included. It is mostly aimed at full system emulation, though (but if you look through the cogs you can also download chroots). https://people.debian.org/~gio/dqib/
Of course, the best long term solution is to use something like yocto or buildroot, but that takes considerable time and knowledge to do properly.
FOSS Unix: 42 steps forward, 42 steps back. :-/
I've had more challenges getting the necessary dependencies installed for foreign architectures than I've had issues running them through qemu. It's honestly surprising how easy it is and it makes me wonder why Windows doesn't have something similar.