switch Raspberry Pi targets to 64bit

[christianrauch]

Solved Problem

Newer Raspberry Pi models (3, 4, 5) use 64bit CPUs and cannot run the 32bit executables generated by arm-linux-gnueabihf.

Changelog Entry

For release notes:

Feature/Bugfix switch Raspberry Pi targets to aarch64-linux-gnu

Context

Related links, screenshot before/after, video

https://www.raspberrypi.com/documentation/computers/processors.html

[oursland]

32-bit is supported by these CPUs, but you need to have a 32-bit ABI enabled. If you choose to install the 32-bit or 64-bit RPiOS, you need to compile for that ABI.

For most users, 32-bit RPiOS is recommended as it greatly reduces memory consumption and improves performance.

[christianrauch]

32-bit is supported by these CPUs, but you need to have a 32-bit ABI enabled. If you choose to install the 32-bit or 64-bit RPiOS, you need to compile for that ABI.

This is not only about Raspberry Pi OS. There are alternative distributions used on the Raspberry Pi, such as Ubuntu, which do not have an "official" 32bit distribution for the Raspberry Pi any more. 32bit also limits you to 4GB RAM.

For most users, 32-bit RPiOS is recommended as it greatly reduces memory consumption and improves performance.

Do you have a source for this? How does running 32bit programs on 64bit hardware improve performance?

[oursland]

Synthetic benchmarks will often show that the 64-bit versions outperform the 32-bit on CPU operations. This neglects the impact of memory. Switching to 64-bit typically increases memory consumption by 50%-100%, depending upon use case (also on page 3 of Phoronix). With values and pointers doubling in size to 64-bits, this means on-chip cache gets an effective 50% reduction in capacity.

Donald Knuth wrote about this in 2008 (A Flame About 64-bit Pointers). The response to this was the creation of the x32 ABI, in which 64-bit features were used with 32-bit pointers on x86-64. I'm unaware of such an ABI for ARM64.

In real-world scenarios, this increase in memory transfers and effective reduction in cache does slow down the operation of the system. Consequently 32-bit outperforms 64-bit, often by more than 10% in real-world behavior. This gets worse if one depends upon swap space, which is more likely under 64-bit architectures.

32-bit limits a developer to 4 GiB of addressable memory within a process, but it does not limit the OS to 4 GiB.

[christianrauch]

Synthetic benchmarks will often show that the 64-bit versions outperform the 32-bit on CPU operations. This neglects the impact of memory. Switching to 64-bit typically increases memory consumption by 50%-100%, depending upon use case (also on page 3 of Phoronix). With values and pointers doubling in size to 64-bits, this means on-chip cache gets an effective 50% reduction in capacity.

Yes, 64 bit consumes more memory. But memory is cheap. You can get a Raspberry Pi 5 8 GB for 20~30 EUR more than its 4 GB version. If memory is a concern, you can double the RAM quite cheaply. But you cannot get a twice as fast CPU for the Raspberry Pi 5 for the same.

Donald Knuth wrote about this in 2008 (A Flame About 64-bit Pointers). The response to this was the creation of the x32 ABI, in which 64-bit features were used with 32-bit pointers on x86-64. I'm unaware of such an ABI for ARM64.

Can we use "-mlong32" for the Raspberry Pi? If we can compile "natively", i.e. use he architecture features, but use 32 bit pointers for the px4 executable, I would also be fine.

In real-world scenarios, this increase in memory transfers and effective reduction in cache does slow down the operation of the system. Consequently 32-bit outperforms 64-bit, often by more than 10% in real-world behavior.

That is one sample point "the 32-bit build is marginally faster when running on a 64-bit machine (~5%)" (Which 10% do you mean?).

I am not saying that 64 bit is always faster. But looking at some more extensive benchmarks

• https://www.phoronix.com/review/raspberrypi-32bit-64bit
• https://www.phoronix.com/review/raspberrypi-os-64bit
• https://www.sevarg.net/2021/12/18/raspberry-pi-32-bit-vs-64-bit/

  For the Pi4, you really should be using a 64-bit OS if there’s no reason not to.

clearly show that 64 bit systems mostly perform faster by a large margin. I also clearly trade 5% decrease in performance on some tasks against 40 % performance improvement on most tasks. I also trade a 40% memory increase against a 40% faster px4 executable.

This gets worse if one depends upon swap space, which is more likely under 64-bit architectures.

Just upgrade your RAM :-)

32-bit limits a developer to 4 GiB of addressable memory within a process, but it does not limit the OS to 4 GiB.

Yes, the limit is per process and not in total per system. But still, why do we want this limit?

[oursland]

Yes, 64 bit consumes more memory. But memory is cheap.

Just upgrade your RAM :-)

You cannot buy more memory for fielded systems. Your comment reads like someone who has never had to maintain thousands to millions of fielded devices.

Your change is for RPi 3, 4 and 5. There is no upgrading the RPi 3 from 1 GiB of RAM, while the default 64-bit RPi OS often exceeds these memory limitations, going straight into swap memory.

Raspberry Pi is still recommending the 32-bit version of their operating system over the 64-bit version, and for good reason. The default PX4 configuration should match the recommended distribution of operating system.

[christianrauch]

Yes, 64 bit consumes more memory. But memory is cheap.

Just upgrade your RAM :-)

You cannot buy more memory for fielded systems. Your comment reads like someone who has never had to maintain thousands to millions of fielded devices.

Well, yes, I haven't. But if you have thousands to millions devices deployed, you won't update them to the current main branch either. You will stick with the stable PX4 and OS version. Do you know of this magnitude of PX4 on Raspberry Pi (Navio2, PilotPi, ...) deployments?

If you are maintaining this magnitude of PX4 on Raspberry Pi deployments, I would be seriously interested to know more about this. I am trying to get PX4 on Raspberry Pi running, but I am questioning myself if this is actually a real use case with applications "in the wild", or if "real" applications use dedicated FC hardware with a MAVLink or other connection to a companion PC.

Your change is for RPi 3, 4 and 5. There is no upgrading the RPi 3 from 1 GiB of RAM, while the default 64-bit RPi OS often exceeds these memory limitations, going straight into swap memory.

I think we are constructing artificial problems here. Can you name a concrete example where someone would go into swapping when changing to a 64 bit OS? Btw, I think swapping is a really bad idea on a critical system like that. I assume most use cases don't use swap. So this question is really about does it fit or not.

I have been affected by hardware deprecation before because the firmware just did not fit into the flash due to new features. But I wouldn't argue that we should stop developing new features because of this. I think it is ok to increase hardware requirements over time to enable new features.

Raspberry Pi is still recommending the 32-bit version of their operating system over the 64-bit version, and for good reason. The default PX4 configuration should match the recommended distribution of operating system.