Cannot build on x64 host

[cbarrete]

Hi,

I've been unsuccessfully trying to build a rpi image from an x64 host for a while using flakes and stumbled upon this repo, which I figured I could use as a starting point.

Unfortunately, nix build fails with a disappointing error: a 'aarch64-linux' with features {} is required to build '/nix/store/p2a6vj6pqk9q6jl2qn8y85l0wfqfmpi7-config.txt.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}. The error makes sense given that this derivation is u-boot related.

Clearly nix is not setup for cross compilation, but I'm not sure which bit might be missing.

I have a setup similar to the one outlined in your readme:

• foreign OS (Debian bookworm) with an up to date Nix 2.13.3
• a nix.conf with flakes enabled and aarch64-linux as an extra platform and the other options for good measure
• qemu setup with binfmt support (as confirmed by the output of update-binfmts --display), installed from the Debian repos
• the nix daemon and basically everything else properly restarted
• using your flake.lock with no alterations
• no nix channels setup, so that cannot accidentally taint any of the versions being used

I'm not expecting support, but I'd be grateful if you could let me know in case the above error message sounds familiar to you, and if so, how you might have handled it in the past.
Also, it'd be extra nice if you could confirm that you can still build this repo using a recent nix version.

Thanks a lot!

[n8henrie]

I bet we can get this sorted out. My primary build machine is Arch, and it's building in CI on Ubuntu. I'll double check tomorrow to make sure it's still working.

Nix is pretty great at cross compilation, but my understanding is that some derivations can be pulled from the binary cache that won't compile locally. Not sure if that's why mine works; I tried a number of configurations to see if anything would be faster or slower, but it always seemed to be able to build.

Are you sure that your nix.conf is being used? I forget the flag, nix show-config or something? It sounds like your extra platforms isn't being respected.

[cbarrete]

Thanks for the quick reply!

I'm fairly certain that my nix.conf is being used:

~ > nix show-config | rg extra
extra-platforms = aarch64-linux

Besides I've gone through the whole "restart the nix daemon service and reboot the machine" cycle more than once.

It sounds like your extra platforms isn't being respected.

It does indeed, but I'm at a loss as to why.

[n8henrie]

I have to set GC_DONT_GC due to a segfault (NixOS/nix#4246), but GC_DONT_GC=1 nix build works fine on my Arch box.

I installed debian bookworm amd64 in an LXC container, and after installing qemu-user-static and setting up my nix.conf (including system-features = kvm -- which I think just fakes it, since /dev/kvm doesn't exist?), it builds without trouble.

Arch:

• system: "x86_64-linux"
• host os: Linux 6.2.7-arch1-1, Arch Linux, noversion, rolling
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.14.1
• channels(root): "nixpkgs-22.11"
• channels(n8henrie): "nixpkgs-22.11"
• nixpkgs: /home/n8henrie/.nix-defexpr/channels/nixpkgs

Debian LXC:

n8henrie@debian:~/git/nixos-btrfs-pi$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.2.7-arch1-1, Debian GNU/Linux, noversion, nobuild`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.3`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

n8henrie@debian:~/git/nixos-btrfs-pi$ nix build
n8henrie@debian:~/git/nixos-btrfs-pi$ echo $?
0
n8henrie@debian:~/git/nixos-btrfs-pi$ ls result/btrfspi.iso.zst
result/btrfspi.iso.zst

I assume you're using the multi-user nix install?

[cbarrete]

Ok so first, thanks a lot for all the help! I've explored lots of things that I probably would not have otherwise.

Second, I can finally build an image, but I don't really know how I got here. I'll list some things that I tried in the hope that it might help someone. I'll eventually try to narrow that to the required subset, but in the meantime, that might provide some inspiration/paths to explore:

1. I had first installed nix from the Debian repositories, but was using a newer version coming from nixpkgs. I had the same failure both as my own user and as root, as well as using either the old /usr/bin/nix or the one from the nix store.
2. nix-info was reporting that I had a single-user install, which was obviously wrong as the nix store was owned by root but I had always been operating as my regular user with builds running as nixbldX users
3. At some point, after fiddling with different configurations in /etc/nix/nix.conf (which seemed to yield different results from ~/.config/nix/nix.conf, but I'm not certain since I was messing with several things at once, not knowing where to look), I got further in the build, but got nasty segfaults in glibc that would kill all my graphical and SSH sessions on that host, both when building as my user and as root. At that point, it seemed that I could get something to build, but the segfaults prevented the builds from finishing. The GC env var made no difference
4. I wiped nix from my system, including /nix, then reinstalled from the official script this time. That brought me back to square one: error: a 'aarch64-linux' with features {} is required....
5. After more tweaking and trying to build, both as root and a regular user, I finally got as root, and later as my regular user, with the following verbatim /etc/nix/nix.conf:

build-users-group = nixbld
cores = 0
experimental-features = nix-command flakes
extra-platforms = aarch64-linux
max-jobs = auto
system-features = kvm

tl;dr:

• try to put the nix settings in /etc/nix/nix.conf, not just your user configuration (I'm doubtful that this has/had any impact)
• try installing nix, especially all the surrounding setup (systemd, groups, users, etc...) via the official script rather than a distro package
• set system-features = kvm on top of the settings suggested in the readme
• something else that I fixed by accident and cannot figure out
• I'll report if I can narrow this down and get the build to both succeed and fail reproducibly

[cbarrete]

Just as I wrote this, I figured something out: extra-platforms = aarch64-linux must be in the system config (and requires a nix daemon restart) and is not picked up in the user-level config.

[n8henrie]

Sweet, glad you got it working.

No idea why having it in the user vs the system config matters, seems to go against the whole point of nix.

Then again, there's plenty I don't yet get in this space 😆