-
Notifications
You must be signed in to change notification settings - Fork 0
preempt_rt_kernel
I tried two methods for cross-compiling, but don't quite have it right yet. I've preserved the attempts here for future reference.
However, native build only took about 45 minutes on a Raspberry Pi 5 with active cooler, so I'm going to set aside cross-compiling.
See:
- Cross compilation https://github.com/danzimmerman/dz_rpi_config/issues/13
- Native compilation https://github.com/danzimmerman/dz_rpi_config/issues/14
- Native compilation but with
rpi-update nextto install 6.12 base kernel https://github.com/danzimmerman/dz_rpi_config/issues/18
The Raspberry Pi OS team is preparing to update to the 6.12 kernel. That means a lot of the basics can be set up more easily:
See https://forums.raspberrypi.com/viewtopic.php?t=379745
We are planning to move to the 6.12 kernel in the near future (perhaps a few months). This is expected to be the next upstream LTS (Long Term Support) kernel.
Still, you have to configure and build the kernel, including downloading the 6.12 sources, see https://lwn.net/Articles/989227/
This allows you to omit this bit I guess:
sudo SKIP_KERNEL=1 PRUNE_MODULES=1 rpi-update rpi-6.12.y
Will keep an eye on this, though. Hopefully there will be a RT option or something in the future?
Followed the instructions exactly at https://github.com/by/RT-Kernel
Transcribing here for future reference.
sudo apt install git bc bison flex libssl-dev make
sudo apt install libncurses5-dev
sudo apt install raspberrypi-kernel-headers
cd ~
git clone --depth 1 --branch rpi-6.12.y https://github.com/raspberrypi/linux
Enter the resulting linux folder and make the basic config:
cd ~/linux
make bcm2712_defconfig
Then enter the menu-driven config
make menuconfig
and select what we need. For a first test, I just set the kernel to fully preemptible and set the frequency governor to performance.
➡️ TO-DO: update with these settings. Transcribed below:
## I've made the following changes specifically for my NTP server to also enable kernel PPS:
-CPU_FREQ_DEFAULT_GOV_POWERSAVE y
-CPU_FREQ_GOV_CONSERVATIVE y
-CPU_FREQ_GOV_ONDEMAND y
-CPU_FREQ_GOV_PERFORMANCE y
-CPU_FREQ_GOV_SCHEDUTIL y
-CPU_FREQ_GOV_USERSPACE y
-LEDS_TRIGGER_CPU y
-NO_HZ y
-PREEMPT y
LOCALVERSION "-v8-16k" -> "-v8-16k-NTP"
PPS_CLIENT_GPIO m -> y
+CPU_FREQ_DEFAULT_GOV_PERFORMANCE y
+CPU_IDLE_GOV_MENU y
+EFI_DISABLE_RUNTIME n
+HZ_1000 y
+NTP_PPS y
+PREEMPT_RT y
+RTC_INTF_DEV_UIE_EMUL y
💡PPS support is probably something I want too. Also setting HZ_1000 is useful for robot controls.
make prepare
make CFLAGS='-O3 -march=native' -j6 Image.gz modules dtbs # recommendation is 1.5 times the number of cores (=4), which equals 6
sudo make -j6 modules_install # recommendation is 1.5 times the number of cores (=4), which equals 6
⏱️This only took about 45 minutes natively.
sudo mkdir /boot/firmware/NTP
sudo mkdir /boot/firmware/NTP/overlays-NTP
➡️ TO-DO: Change the naming in these instructions to reflect better what I'm doing
Add to /boot/firmware/config.txt:
os_prefix=NTP/
overlay_prefix=overlays-NTP/
kernel=/kernel_2712-NTP.img
sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/firmware/NTP/; sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/firmware/NTP/overlays-NTP/; sudo cp arch/arm64/boot/dts/overlays/README /boot/firmware/NTP/overlays-NTP/; sudo cp arch/arm64/boot/Image.gz /boot/firmware/kernel_2712-NTP.img
sudo reboot now
After rebooting:
sudo SKIP_KERNEL=1 PRUNE_MODULES=1 rpi-update rpi-6.12.y
There are some config settings here that may be needed to actually use the RT kernel for ROS 2, etc.
➡️ TO-DO: Research whether bringing PREEMPT_RT into the mainline kernel makes some of the user permissions config unnecessary. I doubt it.
Check the kernel version:
dan@computer:~ $ uname -a
Linux computer 6.12.0-rc3-v8-16k+ #1 SMP PREEMPT_RT Thu Oct 17 09:06:53 EDT 2024 aarch64 GNU/Linux
Add a realtime group and add your user to it:
sudo groupadd realtime
sudo usermod -aG realtime $(whoami)
Edit /etc/security/limits.conf to include:
# For RT kernel and ROS 2
@realtime soft rtprio 99
@realtime soft priority 99
@realtime soft memlock 102400
@realtime hard rtprio 99
@realtime hard priority 99
@realtime hard memlock 102400
Finally do a cyclictest test:
dan@computer:~ $ sudo cyclictest -a -t -p99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 6.76 2.41 0.95 9/397 2740
T: 0 ( 2714) P:99 I:1000 C: 85755 Min: 1 Act: 3 Avg: 2 Max: 15
T: 1 ( 2715) P:99 I:1500 C: 57170 Min: 1 Act: 3 Avg: 2 Max: 12
T: 2 ( 2716) P:99 I:2000 C: 42877 Min: 1 Act: 2 Avg: 2 Max: 11
T: 3 ( 2717) P:99 I:2500 C: 34302 Min: 1 Act: 2 Avg: 2 Max: 13
Default unit is in microseconds, so we see single-digit $\mu$s average and <20$\mu$s maximum. This is with stress --cpu 8 running.
➡️ TO-DO: ROS 2 benchmarks https://ros-realtime.github.io/ros2_realtime_benchmarks/benchmark_tools/cyclictest.html
Following cross-compile instructions here:
https://www.raspberrypi.com/documentation/computers/linux_kernel.html#cross-compile-the-kernel
Doing this on WSL 2, Ubuntu Noble 24.04.1 LTS
- Clone the Raspberry Pi OS kernel sources
git clone --depth=1 https://github.com/raspberrypi/linux
- Prepare for cross-compilation
Install some tools:
sudo apt update
sudo apt install bc bison flex libssl-dev make libc6-dev libncurses5-dev
And get cross-build for ARM64:
sudo apt install crossbuild-essential-arm64
- Patch the kernel
https://www.raspberrypi.com/documentation/computers/linux_kernel.html#patch-the-kernel
See also:
https://github.com/UniversalRobots/Universal_Robots_Client_Library/blob/master/doc/real_time.rst
Check the kernel version:
dan@karman:~/linux$ head Makefile -n 4
# SPDX-License-Identifier: GPL-2.0
VERSION = 6
PATCHLEVEL = 6
SUBLEVEL = 56
We actually want 6.6.53 after 4ad9fa5c30edc19acf05b2960dd686c29cbe75a2
💡 The Raspberry Pi forums don't seem to require an exact match between the kernel version and the RT patch version. Let's try it.
In the cloned linux folder:
wget https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/6.6/patch-6.6.53-rt44.patch.gz
gunzip patch-6.6.53-rt44.patch.gz
cat patch-6.6.53-rt44.patch | patch -p1
This seemed fine, see below.
https://github.com/by/RT-Kernel
In the resulting linux folder run
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2712_defconfig
Then
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
General->Preemption Model->Fully Preemptible
CPU Frequency Scaling -> Default CPUFreq governor (performance)
Then exit out.
From UR Client Library
scripts/config --disable SYSTEM_TRUSTED_KEYS
scripts/config --disable SYSTEM_REVOCATION_KEYS
Then finally:
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j `getconf _NPROCESSORS_ONLN` deb-pkg
Needed also, based on build error:
sudo apt install debhelper-compat cpio libelf-dev
.deb files are created. However, the installation step fails. See here.
This is promising future work but I'm skipping it for now.